Improving the Accuracy of Civil Damage Awards With Claim Aggregation

CitationBavli, Hillel J. 2017. Improving the Accuracy of Civil Damage Awards With Claim Aggregation. Doctoral dissertation, Harvard University, Graduate School of Arts & Sciences.

Permanent linkhttp://nrs.harvard.edu/urn-3:HUL.InstRepos:41142054

Terms of UseThis article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Other Posted Material, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#LAA

Share Your StoryThe Harvard community has made this article openly available.Please share how this access benefits you. Submit a story .

Accessibility

ImprovingtheAccuracyofCivilDamageAwardswithClaimAggregation

Adissertationpresentedby

HillelJ.Bavlito

TheadhocCommitteeforthePh.D.PrograminStatisticsinLawandGovernment

inpartialfulfillmentoftherequirements

forthedegreeof

DoctorofPhilosophy

inthesubjectof

StatisticsinLawandGovernment

HarvardUniversity

Cambridge,Massachusetts

January2017

©2016HillelJ.BavliAllrightsreserved.

iii

DissertationAdvisor:ProfessorDonaldRubin HillelJ.Bavli

ImprovingtheAccuracyofCivilDamageAwardswithClaimAggregation

Abstract

Alegalproceedingcanbeunderstoodasanestimationproblem,wherethequality

ofthelegaloutcomedependsontheinformationusedtoadjudicateit.ThisPh.D.

Dissertationisacompilationoffivepapersthatexamineasetofmethodsforimprovingthe

informationusedtoadjudicatelegaloutcomesinthecontextofcivilclaimsfordamages.

Thepapersanalyzethepossibilityofimprovingcivildamageawardsbyaggregating

informationacrossdifferentclaims—first,acrossclaimsintheclass-actioncontext,and

second,acrossclaimsinaltogetherseparatecasesovertime.Thepapersexaminesuch

methodsstatistically,analyzingwhether,andunderwhatconditions,aggregating

informationregardingdamageawardsorfactsinseparateclaimswouldimprovethe

qualityofawards.Andtheyexaminesuchmethodslegally,assessingwhether,andunder

whatconditions,suchmethodsarepermissibleunderevidentiary,procedural,and

constitutionalframeworks.Thefinalpaperreportsandinterpretsresultsfromafactorial

experimentdesignedtotestcertainconclusionsrelatingtothesemethods.Insummary,the

papersdemonstratethatthemethodsdiscussedarecapableofimprovingdamageawards

substantially,andaregenerallypermissibleunderthelaw.

iv

TABLEOFCONTENTSINTRODUCTION 1

I.AGGREGATINGFORACCURACY:ACLOSERLOOKATSAMPLINGANDACCURACYINCLASSACTIONLITIGATION 4

II.SAMPLINGANDRELIABILITYINCLASSACTIONLITIGATION 49

III.THELOGICOFCOMPARABLE-CASEGUIDANCEINTHEDETERMINATIONOFAWARDSFORPAINANDSUFFERINGANDPUNITIVEDAMAGES 68

IV.SHRINKAGEESTIMATIONINTHEADJUDICATIONOFCIVILDAMAGECLAIMS 114

V.GUIDINGJURORSWITHPRIOR-AWARDINFORMATION:ARANDOMIZEDEXPERIMENT 152

1

Introduction

Alegalproceedingcanbeunderstoodasanestimationproblem.Imaginethatthere

issome“correct”outcome,oratleastarangeof“acceptable”outcomes,thatreflectsthe

policyobjectives(suchasfairnessordeterrence)underlyingthesubstantivelawgoverning

thecaseathand,aswellasthe“true”stateoftheworld,includingthelawandthe

circumstancesofthecase.Aproceeding,suchasahearingortrial,isnecessary,because

thereisimperfectinformationregardingthelawandthecircumstancesofthecase.The

proceedinggeneratesinformationthrough,forexample,argumentationandthe

presentationofevidence.Andbasedonthislimitedinformation,itproducesalegal

outcome—anestimateofthe“correct”outcome.1Thequalityofthelegaloutcome,

therefore,dependsontheinformationusedtoadjudicateit.

ThisPh.D.Dissertationisacompilationoffivepapers,correspondingtothefive

sectionsbelow,thatexamineasetofmethodsforimprovingtheinformationusedto

adjudicatelegaloutcomesinthecontextofcivilclaimsfordamages.Specifically,thepapers

hereinanalyzethepossibilityofimprovingcivildamageawardsbyaggregating

informationacrossseparateclaims.Suchmethodsgiverisetoarangeofstatisticaland

legalquestions.Statistically,thepapersexaminewhether,andunderwhatconditions,

aggregatinginformationregardingdamageawardsorfactsinseparateclaimswould

improvethequalityofawards.Legally,thepapersassesswhether,andunderwhat

1Seeinfra§I.

2

conditions,suchmethodsarepermissibleunderevidentiary,procedural,andconstitutional

frameworks.

Thefirstpaper,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyin

ClassActionLitigation(publishedinLaw,Probability&Risk),2discussesaggregation

proceduresintheclassactioncontext,wherebydamageawardsforclassclaimsare

extrapolatedfromasmallsampleofrepresentativeadjudications.Itexaminesthe

conditionsunderwhichsuchprocedureswouldimprovethequalityofawards,aswellas

relevantlegalconsiderations.Thesecondpaper,SamplingandReliabilityinClassAction

Litigation(publishedinCardozoLawReviewdenovo),3isashortpaperthatexplainsthe

conclusionsinthefirstpaperinnon-mathematicaltermsandextendstheminlightofthe

recentSupremeCourtdecisioninTysonFoods,Inc.v.Bouaphakeo.4

Thethirdpaper,TheLogicofComparable-CaseGuidanceintheDeterminationof

AwardsforPainandSufferingandPunitiveDamages(forthcomingintheUniversityof

CincinnatiLawReview),5analyzesasetofaggregationmethodsapplicableinthe

individual-claimcontext.Whereasthefirstsetofaggregationmethodsappliestotheclass

actioncontext,andthereforeinvolvesapredefinedclassofclaimsarisingfromcommon

factsandissues,thesemethodsinvolvethesharingofinformationacrossaltogether

separatecasesovertime.Specifically,thepaperexaminestheuseof“comparable-case

2HillelJ.Bavli,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyinClassActionLitigation,14LAW,PROBABILITY&RISK67(2015).

3HillelJ.Bavli,SamplingandReliabilityinClassActionLitigation,2016CARD.L.REV.DENOVO207(2016).

4136S.Ct.1036(2016).

5HillelJ.Bavli,TheLogicofComparable-CaseGuidanceintheDeterminationofAwardsforPainandSufferingandPunitiveDamages,U.CIN.L.REV.(forthcoming,2017).

3

guidance”—informationregardingawardsinpriorcomparablecasesprovidedtothetrier

offactasguidancefordeterminingdamageawards—asamethodforimprovingthequality

ofawardsforpainandsufferingandpunitivedamages.Thepaperarguesthatsuch

methodsareimportantandpermissibleunderthelaw,anditappliesbasicstatistical

modelingandtechniquestoexplainhowandwhensuchmethodscanbeexpectedto

improvedamageawards.Thefourthpaper(coauthoredwithYangChen),Shrinkage

EstimationintheAdjudicationofCivilDamageClaims(forthcomingintheReviewofLaw&

Economics),6assumesthatthecomparable-caseguidancemethodsdiscussedabovecould

beappliedformulaically(or,equivalently,assumesthatthetrieroffactwouldincorporate

theinformationaccordingtoourmodel)andderivesarangeofstatisticalresultsregarding

theconditionsunderwhichsuchmethodswouldimprovethequalityofdamageawards.

Finally,thefifthpaper(coauthoredwithReaganRose),GuidingJurorswithPrior-

AwardInformation:ARandomizedExperiment,7reportsandinterpretstheresultsofa

factorialexperimentdesignedtotesttheconclusionsinthethirdandfourthpapers—and

specifically,totestthebehavioralassumptionsinthosepapersandwhethercomparable-

caseguidanceimprovesthequalityofawards(asdefinedinthepapersbelow)undera

robustrangeofconditions.8

Insummary,thepapersdemonstratethatthemethodsdiscussedarecapableof

improvingdamageawardssubstantially,andaregenerallypermissibleunderthelaw.

6HillelJ.Bavli&YangChen,ShrinkageEstimationintheAdjudicationofCivilDamageClaims,REV.OFL.&ECON.(forthcoming,2017).

7HillelJ.Bavli&ReaganRose,GuidingJurorswithPrior-AwardInformation:ARandomizedExperiment.

8Note,thepapershavebeenmodifiedfromtheiroriginalformforpurposesofconsistency.

4

I. AggregatingforAccuracy:ACloserLookatSamplingandAccuracyin

ClassActionLitigation*

1. Introduction

ProfessorCharlesAlanWrightremarkedthat“[u]nlesswecanusetheclassaction

anddevicesbuiltontheclassaction,ourjudicialsystemisnotgoingtobeabletocopewith

thechallengesofthemassrepetitivewrong.”1Indeed,UScourtsincreasinglygrapplewith

thelimitsofaggregationproceduresinmanagingtoday′slitigationneeds.Amongthemost

controversialofsuchproceduresistheuseofstatisticalsampling,andinparticular,

representative“bellwether”trials,toextrapolateaggregateproofofliabilityanddamages

foraclassofunlitigatedclaims.

In1990,withfederalasbestosfilingsaveraging1,140permonth—doubletherateat

whichsuchfilingswerebeingresolved—U.S.courtssearchedfornovellegalprocedures

* Thisarticleoriginallyappearedin14LAW,PROBABILITY&RISK67(2015).Citation:HillelJ.Bavli,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyinClassActionLitigation,14LAW,PROBABILITY&RISK67(2015).Itmayhavebeenmodifiedfromitsoriginalformforpurposesofconsistency.Indevelopingtheideaspresentedinthecurrentarticle,IbenefittedimmenselyfromtheadviceandguidanceofProfessorsDonaldRubinandAlanZaslavsky,andfromthethoughtfulcommentsofananonymousreviewer.Othermajorcontributors,towhomIoweparticulargratitude,includePengDing,RuobinGong,CarlMorris,JoeBlitzstein,KathrynSpier,DavidRosenberg,LouisKaplow,andthestudentsofDonaldRubin′sgraduateseminarcourseatHarvard.IowethankstotheDepartmentofStatisticsatHarvardUniversityforitssupportandtotheLaw,Probability&RiskEditorialBoardfortheirhelpfulcommentsandedits.

1Cimino, etal. v.Raymark Industries, Inc., etal., 751F. Supp.649,652 (E.D.Tex.1990) (citingH.Newberg,NewbergonClassActions17.06,at373(2ded.1985)).

5

capableofhandlingtheinfluxofclaimswhilemaintainingproceduraldueprocessand

ultimatelyajustresolutionoftheclaims.2Inthiscontext,controversystirredovertheuse

ofaggregationmechanisms,suchassampling,toresolvetheclaims.3

InCiminov.Raymark,4a1990asbestosclassaction,JudgeParkeroftheEastern

DistrictofTexasapprovedatrialplaninvolvingarandomsampleof160claimantsfroma

totalof2,298classmembers.Hedividedthepopulationofclassmembersintofivedisease

categoriesbasedontheclaimants′injuryclaims.Hethentookarandomsamplefromeach

categorytoformarepresentative160-claimantsamplegroup.Thedamagesclaimsofeach

randomlysampledclassmemberweresubmittedtoajury,andeachmemberofthesample

groupwasawardedhisindividualizedverdict.Then,theaverageverdictineachdisease

categorywascomputedandappliedtothenon-sampledmembersofeachcategory,

respectively.5TheFifthCircuitultimatelyrejectedthesamplingprocedure,rulingthatit

violatedthedefendant′sSeventhAmendmentrighttoindividualizeddeterminationsof

whethertheproductsatissuecausedplaintiffs′injuries,aswellasitsrighttohavejury

2MichaelJ.Saks&PeterDavidBlanck,JusticeImproved:TheUnrecognizedBenefitsofAggregationandSamplingintheTrialofMassTorts,44STAN.L.REV.815,815-19(1992).

3Anadhoccommittee,createdbytheChiefJusticeoftheU.S.SupremeCourttoexaminepotentialsolutionstothe enormous backlog caused by the asbestos litigation, found that the problem had “reached criticaldimensions,”andthatthecourtsareill-equippedtohandlethesituation.Id.at816(citingJudicialConferenceoftheUnitedStates,ReportoftheJudicialConferenceAdHocCommitteeonAsbestosLitigation,at2(1991)).Amongthemanyrecommendationsmade in itsreport, theCommitteeproposedthepossibilityofsamplingasbestosclaimsforlitigationandthenextrapolatingtheoutcomesoftheremainingclaimsfromthoseofthesampledclaims.

4751F.Supp.649(E.D.Tex.1990).

5Seeid.at652-54.

6

determinationsregardingthe“distinctandseparableissuesofactualdamagessufferedby

eachextrapolationplaintiff.”6

Thesentimentinthecourtsandintheliterature—thenandnow—isthattheuseof

samplingproceduresservespurposesofefficiency,butonlyatthecostoffairness,or

accuracy;andthatthe“benefitsofefficiencycanneverbepurchasedatthecostof

fairness.”7

Butina1992StanfordLawReviewarticle,ProfessorsMichaelSaksandPeter

Blancksoughttoaddresstheconstitutionalandreliabilityconcernsthatcourtsand

commentatorshavevoicedinopposingtheuseofsampling.Theydidnotsimplyarguethat

properlyimplementedaggregationcanprovidethesamelevelofreliabilityas

individualizedlitigation.Rather,theyclaimedthatinfact“aggregationaddsanimportant

layerofprocesswhich,whendonewell,canproducemorepreciseandreliable

outcomes”—that“theproceduralinnovationofaggregationprovidesaqualityofjustice

thatsurpasseswhatcourtshave,untilnow,beencapableofinanykindofcase.”8

6Cimino,etal.v.RaymarkIndustries,Inc.,etal.,151F.3d297(5thCir.1998).

7Malcolmv.NationalGypsumCo.,995F.2d346(2dCir.1993).SeeInreChevronU.S.A.,109F.3d1016,1023(5thCir.1997)(Jones,J.,concurring)(“asJudgeHigginbothamcautionedinInreFibreboardCorp.,thereisafinelinebetweenderivingresultsfromtrialsbasedonstatisticalsamplingandpurelegislation.Judgesmustbesensitivetostaywithinourproperboundsofadjudicatingindividualdisputes.WearenotauthorizedbytheConstitutionorstatutestolegislatesolutionstocasesinpursuitofefficiencyandexpeditiousness.”).InHilaov.EstateofMarcos,acontroversialdecisioninwhichtheNinthCircuitapprovedoftheuseofsamplingtoresolvethe damages claims of approximately 10,000 claimants, the court nevertheless emphasized that “[t]hestatisticalmethodusedbythedistrictcourtobviouslypresentsasomewhatgreaterriskoferrorincomparisontoanadversarialadjudicationofeachclaim.”103F.3d767,786(9thCir.1996).

8Saks&Blanck,supranote2,at815-16.

7

SinceCimino,theneedforeffectiveaggregationprocedureshasonlyintensified.But

assertionsbySaksandBlanck,aswellasotherauthors,thatsamplingcanincrease

accuracyaswellasefficiency,havebeenmetwithskepticismandageneralunwillingness

torelyonsuchassertionsinrulingonproposedaggregationprocedures.Instead,courts

havecontinuedtorejectsamplingonthegroundsthataccuracyshouldnotbesacrificedfor

thesakeofefficiency.Indeed,althoughcourtsandcommentatorshaveoftenrejected

samplingexplicitlyongroundsofprotectingthedefendant′sconstitutionalandprocedural

rights,9SaksandBlanckarelikelycorrectinremarkingthat“amajor—perhapsthemajor—

dueprocessconcerninanaggregatedtrialisthevalidityoftheoutcome.”10AsSaksand

Blanckemphasized,andcourtshavesuggested,“Themainargumentagainsttrialby

aggregationandsamplingassertsthatsuchtrialscannotgivethepartiesasaccuratea

resultastheywouldobtainthroughtraditionalbilateraltrials.”11

9See,e.g.,Wal-MartStores,Inc.v.Dukes,etal.,131S.Ct.2541,2561(2011)(rejectingsamplingongroundsthattheRulesEnablingActprohibits interpretingRule23 so as to “abridge, enlargeormodify any substantiveright.”);Ortizv.FibreboardCorp.,527U.S.815,845(1999)(citingAmchemProducts,Inc.v.Windsor,521U.S.591,613(1997)).

10Saks&Blanck, supranote2, at 833. Saks andBlanck explain the instrumentalist argument that theDueProcessClauseexists toservepurposesofaccuracy—that “[t]herights tonotice,hearing,andcounseleachcontributetothegoalofaccuracy.”Id.(citingJointAnti-FascistRefugeeComm.v.McGrath,341U.S.123,172(1951)(Frankfurter,J.,concurring));MartinH.Redish&LawrenceC.Marshall,AdjudicatoryIndependenceandtheValuesofProceduralDueProcess,95YALEL.J.455,476-77(1986).TheycommentthattheSupremeCourt′sbalancingapproachinMathewsv.Eldridge,424U.S.319,334-35(1976),“attemptstoensuretheaccuracyofoutcomefromtheprocess.Itisrelativelylessfocusedonthefairnessorperceivedfairnessoftheproceduresthemselves.” Saks&Blanck, supranote2, at 827 (citing JerryL.Mashaw,The SupremeCourt′sDueProcessCalculusforAdministrativeAdjudicationinMathewsv.Eldridge:ThreeFactorsinSearchofaTheoryofValue,44U.OFCHICAGOL.REV.28(1976)).

11Saks&Blanck,supranote2,at833.SeealsoInreChevronU.S.A,Inc.,109F.3dat1020(“ourproceduraldueprocessconcernsfocusonthefactthattheprocedureembodiedinthedistrictcourt′strialplanisdevoidofsafeguardsdesignedtoensurethattheclaimsagainstChevronofthenon-representedplaintiffsastheyrelatetoliabilityorcausationaredeterminedinaproceedingthatisreasonablycalculatedtoreflecttheresultsthatwouldbeobtainedifthoseclaimswereactuallytried.”).

8

Arguably,theprimarysourceofsuchskepticism,andtheunwillingnessofcourts

andcommentatorstorelyonassertionsthatsamplingcanincreaseaccuracyaswellas

efficiency,isthegeneralitywithwhichsuchassertionshavebeenmade.Inparticular,such

assertionshavenotbeendevelopedformally,andtheliteraturehasnotexaminedtheeffect

ofsamplingonaccuracy,inanyrigorousform,underconditionsapplicabletoreal-world

classactions.Indeed,inmostcircumstances,samplingdoesnotincreaseordecrease

accuracycategorically;rather,asexplainedbelow,itscapacitytoincreaseaccuracyisa

functionoftheparametersassociatedwithaparticularclassofclaims.Therefore,adeeper

understandingoftheconditionsunderwhichsamplingwillincreaseaccuracyisnecessary.

Inthecurrentarticle,Iintroduceaframeworkforexaminingtheconditionsunder

whichsamplinginthecontextofclassactionlitigationcanincreaseaccuracyinthelaw.In

particular,Idevelopamodelforstudyingtheeffectsofsamplingonaccuracyandfor

derivingtheoptimalsamplesize,givenasetofparametersassociatedwithaclassof

claims.Ithenintroducemethodsforestimatingclassparametersandforincreasingthe

utilityofclaimaggregation.

IbegininSection2bymotivatingtheproblemwithabriefoverviewofsamplingin

thecontextofclassactionlitigation,andadiscussionregardingtheimportanceof

understandingtheconditionsunderwhichsamplingcanincreaseaccuracy.InSection3,I

discussinfurtherdetailthemeaningofaccuracyandalsotheaspectofthelawthat

underliestheaccuracy-enhancingbenefitsofsampling:judgmentvariability.InSection4,I

introduceabasicframeworkforexaminingtheeffectsofsamplingonaccuracy,andfor

9

derivingtheoptimalsamplesize,givenclassparametersandspecifiedconstraints.In

Section5,Idiscussanumberofimportantextensions,includingpossibilitiesforparameter

estimationandtheuseofsequentialsampling,stratificationtechniquesthatmayallowfor

thereductionofclaimheterogeneityandtherebyincreasetheutilityofclaimaggregation,

andissuesrelatedtoincentivestoinitiateclaimsandtosettleundercircumstancesofclaim

aggregation.InSection6,Iconclude.

2. AggregateProofandSamplinginClassActionLitigation

Aclassactionisalegalprocedurebywhichmultiple(oftenmany)legalclaimsare

aggregatedintoasinglecaseforjointdetermination.Classactionsconstituteavery

substantialproportionofclaimsintheUSA.Efficiencyistheprimarypurposeoftheclass

actionprocedure,withoutwhichcourtswouldbeoverwhelmedwithclaimsbroughtfor

individualizedlitigation(leadingtoextremedelays,etc.),andmanyclaimantswith

legitimateclaimswouldbeunabletoaffordbringingtheirclaimsindividuallyinthefirst

instance.

Beforeacourtwillapprove,or“certify,”aproposedclassforthisformoflitigation,

theclassrepresentativesmustprovetothecourtthattheclaimsinvolvecommon

questionsoffactorlaw,andthatitmakessensetotrytheclaimstogetherasaclassaction

ratherthanindividually.Therepresentativesmustshowthattheclaimsaresufficiently

numeroustojustifytheclassactionprocedure;thattheclaimsintheproposedclassare

sufficientlycommon;thattherepresentativeclaimsaretypicaloftheremainingclaims;and

thattherepresentativescanadequatelylitigatetheseclaimsonbehalfoftheclass.In

10

addition,theproposedclassmustsatisfyoneofthreeotherrequirements—frequently,that

commonissuespredominateoverindividualissues.

Itisanarduousprocesstoachieveclasscertification.Inasense,itisatrialinitself

todeterminewhetheraproposedclassshouldbecertified.Afterall,iftheclassisnot

certified,mostclaimantsareunlikelytobringtheirclaimsindividually;andiftheclassis

certified,theclassactionhasthepotentialtobankruptevenlargecompanies,andthe

claimantswillhaveenormouspoweroverthedefendant(s)inelicitingafavorable

settlement.

Anaspectofclasscertificationthathasprovenparticularlydifficultforplaintiffsis

therequirementthataplaintiffclassproveliabilityanddamagesonaclasswidebasis.As

theSupremeCourtheldinWal-Martv.Dukes,whatisimportantindeterminingwhetherto

certifyaclassisnotsimplytheraisingofcommonquestions,butthecapacityoftheclass

proceedingtoproducecommonanswerscentraltothelitigation.Importantly,anindividual

plaintiffmaynotescapetherequirementsofprovingliabilityorinjurymerelybyjoininga

class.Plaintiffsmustbeabletoshowthatthedefendantisliablewithrespecttoeach

claimant,andthatthedefendant′sconductcausedinjurywithrespecttoeachclaimant.

Itishere,inthecontextofprovingclasswideliabilityanddamages,thatclass

plaintiffshaveattemptedtoemploysamplingproceduresandbellwethertrials.12

12“Bellwethertrials,”or“testcases,”havebeenusedinvariouslegalcontexts—suchasantitrustandmass-tortlitigation—andforarangeofpurposes.Asdiscussedfurtherbelow,however,courtshavebeenreluctanttoapprove suchprocedures for purposes of determining liability anddamages. Arguably, prior to a string ofnegative court rulings in andafter2011, therewas risingmomentum for theuseof, or at least for furtherexploration of, such procedures in the mass-tort context. See generally Laura E. Ellsworth & Charles H.Moellenberg, Jr.,Bellwether Trials, 2 Business and Commercial Litigation in Federal Courts §14:47, 3d ed.(2013);Manual forComplexLitigation§22.315,TestCases,Federal JudicialCenter,4thed.(2004);EldonE.

11

Sampling-basedclaimaggregationhastakentwogeneralforms.Thefirstform

involvesaggregatingandsamplingevidencepriortotrial—e.g.,evidencethatthe

defendant′sbehaviorcausedtheharmallegedbytheplaintiffs—followedbythe

presentationofthesampledevidencetoajury,whichwillthendetermineanoutcomefor

theentireclass.13Forexample,inHilaov.EstateofMarcos,aclassactionseekingdamages

forhumanrightsabuses,thedistrictcourtusedarandomsampleof137claims,divided

intocategories(torture,summaryexecutionanddisappearance),fromatotalof

approximately10,000claims,todetermineappropriatecompensatorydamages.14The

districtcourtappointedaspecialmastertosupervisethedepositionsinthePhilippinesof

the137sampledclaimantsandtheirwitnesses.Thespecialmasterthenreviewedtheclaim

formsandthedepositions,evaluatedthetestimonybasedoninstructionsprovidedbythe

districtcourt,andmaderecommendationsregardingthevalidityoftheclaims.He

recommendedthat6ofthe137claimsinthesamplebefoundnotvalid;and,forthe

Fallonetal.,BellwetherTrials inMultidistrictLitigation,82TULANEL.REV.2323(2008);AlexandraD.Lahav,BellwetherTrials,76GEORGEWASH.L.REV.576(2008);JohnC.Coffee,Jr.,ClassWars:TheDilemmaofTheMassTortClassAction,95COLUM.L.REV.1343(1995);DeborahR.Hensler&MarkA.Peterson,UnderstandingMassPersonalInjuryLitigation:ASocio-LegalAnalysis,59BROOKLYNL.REV.961(1993).

13SeegenerallyLaurensWalker&JohnMonahan,SamplingLiability,85VIRGINIAL.REV.329(1999).Notethattheuseofsamplingandstatisticalanalysisinthegeneralevidentiarycontexthasalongandwell-establishedhistory inU.S. courts.See generally DanaG.Deaton,TheDaubert Challenge to the Admissibility of ScientificEvidence, 60 AM. JURISPRUDENCE TRIALS 1 (2014); Crashworthiness Litigation, 2d (AAJ Press) §22, ExpertTestimony:TheEffectoftheDaubertProgeny(2013);R.ClaySprowls,TheAdmissibilityofSampleDataintoaCourtofLaw:ACaseHistory, 4UCLAL.REV.222 (1956-1957);ShariSeidmanDiamond,ReferenceGuideonSurveyResearchinReferenceManualonScientificEvidence359-423,3ded.,FederalJudicialCenter/NationalAcademy of Sciences (2011) (citing cases and literature); Joseph L. Gastwirth, Reference Guide on SurveyResearch,36JURIMETRICSJ.181(1996)(reviewingDiamond′sReferenceGuideonSurveyResearchinReferenceManualonScientificEvidence(1994),anddiscussingrelevantcasesinthetrademark-infringementcontext).

14103F.3d767,782-84(9thCir.1996).

12

remaining131validclaimsinthesample,hemadedamagesrecommendationsforeachof

thethreecategories.Finally,hemaderecommendationsondamageawardsforthe

remaining(non-sampled)classmembersbasedonhisrecommendationsforthesampled

claims.Inparticular,basedonhisfindingthat6ofthe137claims(4.37%)wereinvalid,he

recommendeda5%“invalidityrate”fortheremainingclaims,andherecommendedan

awardfortheclassdeterminedbymultiplyingthenumberofclaimsineachsubclass,after

discountingbytheinvalidityrate,bytheaverageawardrecommendedfortherandomly

sampledclaimsineachsubclass,respectively.Ajurytrialoncompensatorydamageswas

thenheld.Anexperttestifiedregardingthevalidityofthesamplingmethodologyandthe

useofinferentialstatistics,andtestimonyfromthe137sampledclaimantsandtheir

witnesseswasintroduced.Then,thespecialmastertestifiedregardinghis

recommendations,andhisreportwasprovidedtothejury.Thecourtinstructedthejury

thatitcouldaccept,modifyorrejecttherecommendations,andthatitcanreachitsown

judgmentastodamagesonthebasisoftheevidenceofthesampledclaimants.After

deliberatingfor5days,thejuryfoundagainstonly2ofthe137claimantsinthesample,but

generally(withsomeexceptions)adoptedthespecialmaster′srecommendations.The

districtcourtthenenteredjudgmentfor135ofthe137claimantsintheamountsthejury

awarded,andenteredjudgmentfortheremainingclaimantsineachofthethreesubclasses

intheamountsthejuryawarded,tobedividedprorata.15

15Id.;seealsoBlueCrossandBlueShieldofNewJersey,Inc.v.PhilipMorris,Inc.,178F.Supp.2d198(E.D.N.Y.2001)(Individualactionbyhealthinsureragainsttobaccocompaniesinwhichasampleof156deponentswasselected,andstatisticalevidencewaspresentedtothejuryincombinationwithstudies,relevantportionsofvideotapeddepositiontestimony,andexperttestimonytoenablevalidstatisticalinferencetoprovecausation-todeterminewhethersmokerswereinfactdeceivedbystatementsmadebythetobaccoindustry);Walker&

13

Thesecondform—thefocusofthecurrentarticle—involvestheuseof

representative“bellwethertrials”toresolveclasswideissues.Theprocedureapprovedby

JudgeParkerinCiminov.Raymarkisillustrative.InCimino,aclassofasbestosclaimswas

dividedintofivediseasecategories,andasmallrandomsampleofclaimswasselectedfrom

eachcategoryforindividualizedjurydetermination.16Thecourtawardedthesample-

groupclaimantstheirrespectiveindividualizeddeterminations,andthentheaverage

awardineachdiseasecategorywascomputedandawardedtotheremaining(non-

sampled)claimantsineachcategory,respectively.17

ConsidertheimportantcaseWal-Martv.Dukes,18aclassactionlawsuitby

approximately1.5millioncurrentandformerfemaleemployeesagainstWal-Martalleging

thatthesuperchainstorediscriminatedagainstthemonthebasisofgender.Theplaintiffs

arguedthat,inordertoshowclasswideliability,theymustshowthatWal-Martindeed

discriminatedagainstwomen,andthatthe1.5millionclassmemberswerefemaleswho

wereemployedbyWal-Martduringtherelevantperiod.Theyarguedthatthisissue-

Monahan,supranote13(discussingsampling in thetobacco litigationandEstateofMarcos,anddescribingcases leading up to Estate of Marcos). See generally In re Fibreboard Corp., 893 F.2d 706 (5th Cir. 1990)(rejectingthe“lumpsum”approach,inwhichevidencefromselectedtypicalplaintiffsprovidethebasisofthejury′sdeterminationofadamagesawardfortheentireclass).

16Methods for selecting bellwether trials can be categorized as follows: “(i) plaintiffs′ counsel selects thesample;(ii)eachsideselectsanequalnumberofsamplecases;(iii)thecourtselectsthesamplefromnomineesproposedbytheparties;or(iv)thesamplesarerandomlyselectedfromagreed-oncategoriesofclaimants.”2McLaughlinonClassActions8:6(10thed.).SeeSterlingv.VelsicolChemicalCorp.,855F.2d1188,1196n.6(6thCir.1988).Hereafter,unlessstatedotherwise,thereadershouldassumethatclaimsareselectedforthesamplegrouprandomlyfromtheclass,orfromclassstrataiftheclassisstratified,suchasinCimino.

17SeegenerallyCimino,751F.Supp.at652-54.Note,althoughCiminoinvolvedsamplingasameansofresolvingdamagesissues,hereinIconsidersamplinginabroadercontext—includingitsusetoresolveissuesofliability.

18131S.Ct.2541,180L.Ed.2d374(2011).

14

whetherWal-Martdiscriminatedagainstwomen-isthecommonquestionthatbindsall

membersoftheclass,andthatmakesaclassaction,ratherthanindividualizedlitigation,

appropriate.

Thedefensecounteredthat,asamatteroflaw,ifanemployerissuedongroundsof

discriminationthenirrespectiveofwhethertheemployerwasdiscriminatoryornot,the

employermayputforththedefensethattheparticularconductallegedbyagivenplaintiff

wascausedbylegitimate,ratherthandiscriminatory,purposes(e.g.,thataparticular

employeestolefromthecashregisterandwasfiredonthosegroundsalone).Itarguedthat

thisaffirmativedefensemustapplyonanindividualizedbasis,ratherthanthroughsome

shortcutthatmaybeappliedcommonlytotheclassasawhole.

Thelowercourtsapprovedatrialplanthatwasintendedtoovercometheproblem

describedbythedefendants.Theplaninvolvedtakingasmallsampleofclaims(the

“samplegroup”)thatwouldbeexaminedtodeterminewhatproportionofthesample

wouldresultinadeterminationofWal-Mart′sliability.Further,damageswouldbe

determinedforeachofthesampledclaims.Theclaimsinthesamplegroupwouldreceive

individualizeddeterminations.Butthecourtwouldthenassigndamagestotheremainder

oftheclass(the“extrapolationgroup”)byaveragingthedamagedeterminationsfromthe

samplegroup(includingzerodamagesforclaimsinwhichWal-Martwasfoundnotliable)

andmultiplyingthataveragebythenumberofremainingclaims.

TheSupremeCourtreversedthedecisionofthelowercourtsandoutrightrejected

theuseofsamplingtoproveclasswideliabilityanddamageswheresuchsamplingabridges

15

therightofadefendanttodefenditselfonanindividualizedbasisunderthesubstantive

law.TheSupremeCourtviewedthetrialplanasaformof“TrialbyFormula”and

“disapprove[dof]thatnovelproject.”

Wal-Martreaffirmedandperpetuatedasignificantbodyofcaselawrejectingthe

useofsamplingasameansofofferingaggregateproofofclasswideliabilityanddamages.19

InthewakeofWal-Mart,anunderstandingofconditionsunderwhichsamplingcan

increaseaccuracyaswellasefficiencyisoffundamentalimportance.ProfessorWrightwas

likelycorrectwhenhewarnedofthenecessityofbuildingontheclassactionprocedureto

addressthechallengesofthemassrepetitivewrong.20Arguably,inlightofthenatureof

today′ssocialandeconomicinteraction,theneedforsuchprocedureshasneverbeen

greater.Thecourtsareneverthelessincreasinglyunwillingtobenefitfromtheefficiencies

ofaggregation,butunderthefalsepremisethatsuchbenefitscomeonlyatthecostof

accuracy.Butperhapsthegreatestbenefitsofaggregationinfactlieinitsabilitytoincrease

accuracyinadditiontoefficiency.

AsSaksandBlancksuggested,thevalidityofthecase′soutcomeisamajor(ifnotthe

major)sourceofcourts′antagonismtowardssampling,includingantagonismexpressedin

19See,e.g.,McLaughlinv.AmericanTobaccoCo.,522F.3d215,224(2dCir.2008);Cimino,151F.3dat297;InreFibreboardCorp.,893F.2dat706;Bascov.Wal-MartStores,Inc.,216F.Supp.2d592(E.D.La.2002).SeealsoEspenscheidv.DirectSatUSA,LLC,705F.3d770(7thCir.2013)(holdingthatplaintiffs′proposedtrialplanwasnot feasibleduetovarianceindamagesacrossclassmembers, thusrejectingplaintiffs′proposaltoaddresssuchvariancebypresentingtestimonyattrialfrom42representativemembersoftheclass).ButseeSaks&Blanck,supranote2,at837(emphasizingthat“fromtheviewpointofdefendants,eveniftherearerelativelylargeerrors,withnumerousover-andunder-awards,allofthosedifferenceswillcanceleachotheroutandtheaverage award will be the same in the collective trial as it would have been with the individualizeddeterminations").Forcaselawanddiscussion,see2McLaughlinonClassActions,supranote16,§8:7.

20Seesupranote1.

16

termsofcourts′concernfortheeffectofsamplingonadefendant′sproceduralrights.21If

thisisthecase,thenanunderstandingoftheconditionsunderwhichsamplingcanincrease

accuracymayleadcourtstoreversetheircurrentcourse.22

Further,regardlesswhetheraccuracyisatthecenterofjudicialdecisionsrejecting

sampling,anunderstandingoftheaccuracy-enhancingbenefitsofsamplingenablescourts

andlawmakerstobetteraccountforthecostsofsubstantivelawsthatprevent,orthatdo

notenable,suchprocedures.Infact,currently,numerousfederalstatutesexplicitlyor

impliedlypermitsamplingasatoolforprovingdamages.Forexample,theHart-Scott-

RodinoAntitrustImprovementsActstatesthatinparenspatriaeactionsbystateattorneys

generalinwhichtherehasbeenadeterminationthatadefendantagreedtofixprices,

“damagesmaybeprovedandassessedintheaggregatebystatisticalsamplingmethods,by

thecomputationofillegalovercharges,orbysuchotherreasonablesystemofestimating

21SeeSaks&Blanck,supranote2,at833;seealsosupranotes10-11,andaccompanyingtext;2McLaughlinonClassActions,supranote16,§8:7.Note,arguablyWal-Martmaybeviewedasaspecialcase—andarguablydistinguishable from other cases in which a plaintiff class seeks to employ sampling to prove liability ordamages—inthatitwasinWal-Mart′sinteresttodefendagainsttheplaintiffs′claimsonaclaim-by-claimbasis,independentofthevalidityofthedamagesincurredbyWal-Martintheaggregate.Inparticular,itwasinWal-Mart′sinteresttosortvalidclaimsfrominvalidclaimsandtherebysetastrongprecedentforemployeesthattheywillberewardedbasedontheirperformance,andnotbasedonwindfalllegaldamages.Arguably,acourtmayfulfillthissortingfunctionaswellbyorderingpost-awardallocationhearings.

22Note,SaksandBlanckdoaddresstheimportanceofmaintainingproceduraljusticeforitsownsake,buttheyconsidertheproceduralinjusticethatcanresultfromtraditionalproceduresforthemajorityoftheclass,andthey ultimately conclude that, at least with respect to the plaintiffs in actions like the asbestos litigation,aggregationcanprovidemoreproceduraljusticethanthealternatives.SeeSaks&Blanck,supranote2,at838-39.SaksandBlanckarguethat“acloserlookattheaggregatedtrial,atleastinthemasstortcontext,suggeststhat this procedure does not necessarily violate traditional notions of due process under the Fifth andFourteenthAmendments.”Theyassertthat,“[i]nfact,theabsenceofsuchproceduresistantamounttodenyingmany litigants theirdueprocess trial rightsaltogether.” Id. at826.Further, regardlesswhetherproceduraljusticeisvaluedinitsownright,fewwouldarguethatsuchvaluecanbeviewedentirelyindependentlyofitsinstrumentalvalueinensuringaccurateoutcomes,orthat“proceduraljustice”shouldbepreservedregardlessofthepotentialcostswithrespecttoaccuracy.

17

aggregatedamagesasthecourtinitsdiscretionmaypermitwithoutthenecessityof

separatelyprovingtheindividualclaimof,oramountofdamageto,personsonwhose

behalfthesuitwasbrought.”23

Thus,inthefollowingsections,Idevelopaframeworkforexaminingtheconditions

underwhichsamplingcanincreasetheaccuracyoflegaldeterminations.Ibeginby

describingtheconceptsofaccuracyandvariabilityinthelaw.

3. AccuracyandVariabilityintheLaw

Centraltothedebateoversamplingandtotheframeworkdevelopedhereinarethe

conceptsofaccuracyandjudgmentvariability.Ibegin,inthecurrentsection,withageneral

discussionoftheconcepts,andthenintroduceamoreformalconceptualizationinSection4

below.

3.1 TheMeaningofAccuracy

Onerelativelysimpleapproachtodescribingaccuracyinthelawistoassumethat,

foreverylegalclaim,thereexistsa“correct”outcomethatcan,intheory,bedeterminedas

afunctionofthecompletefactssurroundingagivenclaimandthe“true”stateofthelawat

thetimeoftheclaim.24Inthissense,assumingthatcompleteknowledgeofthefactsandthe

2315U.S.C.A.§15d;seealsotheTruthinLendingAct,15U.S.C.A.§1640(a);theEqualCreditOpportunityAct,15U.S.C.A.§§1691etseq.SeegenerallyWalker&Monahan,supranote13.Foradescriptionoftheroleofstatisticsinproving individualcausationanddamages,see2McLaughlinonClassActions,supranote16,§8:7.Note,undercertaincircumstances,anunderstandingoftheconditionsunderwhichsamplingcanincreaseaccuracymayencouragelitigantsthemselvestoagreetotheuseofsampling.

24Amorecomplexapproachistoconsideraprobabilitydistributionof“correct”outcomesassociatedwitheachandeverylegalclaim.

18

stateofthelawisunavailable,atrialwillresultinanestimateofthecorrectoutcome.25As

such,eachlegaloutcomeinvolvesanerrortermthat,ifpossibletocompute,would

describethedistancebetweentheactualoutcomeandthecorrectoutcome.Wecanthen

defineaccuracyastheproximityoftheactualverdicttothecorrectverdict.26

ThisdescriptionisconsistentwithSaksandBlanck′sassertionthatitmakessense

“tothinkofthe‘true’awardastheaverageofthepopulationofpossibleawards”that

wouldresultfromtryingthesamecaserepeatedlyforaninfinitenumberoftimesunder

variousconditions(e.g.,beforedifferentjuries,bydifferentattorneys,usingdifferent

permutationsoffacts,etc.).27Forsimplicity,Iignorethepotentialforbiasandadoptthis

formulationofthe“correct”outcomeassociatedwithagivenclaim.28

Suppose,forexample,thatacigarettesmokerfilesalawsuitagainstamajor

cigarettemanufacturerforfailuretoadequatelyinformtheconsumeroftheharms

associatedwithsmoking.Followingtrial,ajuryreturnsaverdictfindingthecigarette

manufacturerliableandawardstheplaintiffdamagesintheamountof$800,000.Wecan

25Findinganappropriateoutcomecanbeunderstood, instatistical terms,asa “missing-data”problem.SeegenerallyDonaldB.Rubin&RoderickJ.A.Little,StatisticalAnalysiswithMissingData,Wiley,NewYork(2ded.2002).

26SeegenerallyJonathanJ.Koehler&DanielShaviro,VeridicalVerdicts:IncreasingVerdictAccuracyThroughtheUseofOvertlyProbabilisticEvidenceandMethods,75CORNELLL.REV.247(1990)(discussingthemeaningof“accuracy”inthelaw).

27Saks&Blanck,supranote2,at833-34.

28Butseeinfranote41.Note,ifjudgmentvariabilitywereentirelyduetovariabilityinthecompositionofthejury,we couldarrive at an intuitivevariantof SaksandBlanck′s “true” awardby imaginingahypotheticalsituation inwhicheverypossible jury inagivengeographicregionobservesthetrialand independently(ifhypotheticallypossible)determinesaverdict.Thenthe“correct”outcomemaybedescribedastheaverageofallsuchverdicts.

19

assumethat,givencompleteinformationregardingthelawandthefactsofthecase,wecan

computethecorrectoutcome;butsinceperfectinformationregardingthefactsandthelaw

isunavailable,thejuryarrivesataverdict,whichservesasanestimateofthecorrect

outcome.Inourexample,theaccuracyoftheverdictisdefinedastheproximityofthe

verdict,$800,000,tothecorrectoutcome.29

Thus,asdiscussedabove,accuracyisoftenunderstoodasafundamentalobjective

ofthelaw.SaksandBlanckdiscussthisinterestwithinthecontextofwhattheyrefertoas

“distributivejustice”;andtheyhighlighttheinstrumentalistperspectivethat“the

constitutionalpurposeofthedueprocessclauseistoensurethemostaccuratedecision

possible.”30Further,however,maximizingtheaccuracyoflegaldeterminationsislikelyto

servetheaimsofthelaw,whetherornotaccuracy,ordistributivejustice,isitselfthe

primaryobjectiveofthelaw.31Whetheraccuracyisunderstoodasameansofachieving

29 Itmay bemore intuitive to consider the concept of a “correct” verdict in the context of a criminal trial.Consider,forexample,theO.J.Simpsonmurdertrial.Pollsshowthatover50%ofAmericansbelievethatthejuryarrivedatanincorrectverdict.Implicitinthepublic′sdisagreementwiththeverdictisanassumptionthatthereexistsa“correct”outcome.Theframeworkdescribedaboveisintuitive:hadthejuryknownthetruefactsofthecase,andhadtherebeennoambiguityregardingtheapplicationofthelawtothefactsofthecase,thejurywouldhavearrivedatthecorrectconclusion.Butgivenambiguityregardingeitherthefactsofacaseorthe state of the law, it is unclearwhether a criminal defendant in fact satisfied the elements of the crimecharged;andthe jurymustarriveataverdict—“guilty”or“notguilty”—whichservesasanestimateof thecorrect outcome, “guilty” or “not guilty.” Further, given a finding of guilt, the court must determine anappropriate punishment (e.g., an appropriate prison sentence), which can similarly be understood as anestimateofsomecorrectoutcome.Thus,ultimatelythecourtmayarriveataprisonsentencethatservesasanestimateofsomecorrectsentence,whereafindingof“notguilty”translates,forexample,toaprisontermofzero.

30Saks&Blanck,supranote2,at833(citingRedish&Marshall,supranote10,at476-77).

31Ithasbeenargued,forexample,thatthelawshouldaimtomaximizewelfareratherthanfairness.SeeLouisKaplow & Steven Shavell, Fairness Versus Welfare, 114 HARV. L. REV. 961 (2001). Regardless whethermaximizingaccuracyisdeemedtobeafundamentalgoalofthelawinandofitself,thereisastrongargumentthataccuracy(oratleasttheperceptionthereof)isfundamentaltoachievingalternativegoals,suchaswelfare.Forexample,alowdegreeofaccuracyinthelawmayservetodilutethedeterrenceeffectintendedbythelaw;

20

importantlegalobjectives,oritselfacriticalobjectiveoflegalprocedure,theroleof

accuracyinthelawisfundamental.Further,asemphasizedabove,concernforaccuracyis

atleastanimportantfactor—ifnottheprimaryfactor—underlyingthewidespread

rejectionofsampling-basedaggregationinthecontextofclassactionlitigation.

Therefore,throughoutthearticle,Iassumethat,withrespecttosampling,accuracy

isafundamentalobjectiveoflegalprocedure.Further,Iassumethatsampling,ingeneral,

promotesgoalsofefficiency,sincesampling-basedtrialplansarecommonlyofferedasan

alternativetoindividualizedlitigation.However,Ireturntothematterofefficiencyin

Section5.

3.2 Judgmentvariability:ASourceofErrorintheLaw

Theclassactionprocedureallowsacourtsimultaneouslytodecide,withbinding

effect,numerousclaimsinvolvingcommonissues.Atthecostofforegoingindividualized

litigation,theclassactiondeviceoffersarangeofbenefits—primarily,litigationefficiency

andeconomy,andtheprotectionofrightsofindividualswhomaynotbeabletobring

claimsonanindividualbasis.32

andconversely,ahighdegreeofaccuracyinthelawmayservetoenhancesucheffects.Tobesafe,accuracy—evenifnottheprimarygoaloflaw—canbesaidtofacilitatethelaw′sobjectives,suchaswelfare.

32Am.Pipe&Const.Co.v.Utah,414U.S.538,553(1974);Crown,Cork&SealCo.,Inc.v.Parker,462U.S.345,349(1983)(statingthattheprincipalpurposesoftheclassactionprocedurearethe“promotionofefficiencyandeconomyoflitigation”);Andrewsv.ChevyChaseBank,545F.3d570,577(7thCir.2008)(emphasizingthattheprimarypurposesof theclassactionare judicialeconomyandefficiency);Haleyv.Medtronic, Inc.,169F.R.D.643,647(C.D.Cal.1996)(“Classactionshavetwoprimarypurposes:(1)toaccomplishjudicialeconomybyavoidingmultiplesuits;and(2)toprotecttherightsofpersonswhomightnotbeabletopresentclaimsonanindividualbasis.”);InreCaesarsPalaceSec.Litig.,360F.Supp.366,398(S.D.N.Y.1973)(suggestingthattheprimarypurposeof theclassactiondevice is“togivesmall investorsareasonableopportunitytovindicatetheirclaimsinamannerwhichwillnotplaceanunduefinancialburdenuponthem”).

21

Samplinghasbeenusedasameansofpreservingtheclassactionevenwhere

plaintiffsareunabletootherwiseofferclasswideproofofliabilityanddamages.Theideais

todetermineoutcomesthroughindividualizedlitigationforasmallsampleofclaims,and

thentoestimateoutcomesfortheextrapolation-groupclaimsintheformofanaggregate

determinationextrapolatedfromthesample-groupdeterminations.Opponentsof

aggregateproofarguethatwhereclasswideliabilityanddamagescannotbeshownabsent

extrapolationfromasamplegroup,thecourtshouldrejecttheclassaction,e.g.,infavorof

individualizedlitigation,whereineachplaintiffmustbringherclaimindividuallyifatall.

Thatis,opponentsofsamplingarguethatthecourtshouldnotallow“proceduraleconomy

toovershadowsubstantivelaw,”33itshouldnotsacrificeaccuracyforthesakeof

efficiency.34

Butopponentsofsamplinghaveneglectedacriticalfeatureoflitigation:as

highlightedabove,allverdictsare,inasense,estimatesofthetruth;andtryingonecase

beforetwoindependentjuriesishighlylikelytoresultintwodifferentestimatesofthe

correctoutcome.

Intermsofvariability,opponentsofsamplinghavefocusedontheerrorthatresults

fromextrapolation—e.g.,fromapplyingtheaverageofthesample-groupdeterminationsto

33SabyGhoshray,HijackedbyStatistics,RescuedbyWal-Martv.Dukes:ProbingCommonalityandDueProcessConcernsinModernClassActionLitigation,44LoyolaU.ChicagoL.J.467,470(2012).SeegenerallyMalcolm,995F.2dat346.

34Seesupra§2.Foradiscussionofconstitutionalargumentsagainstsampling,seegenerally JosephBarton,Utilizing Statistics and Bellwether Trials in Mass Torts:What Do the Constitution and Federal Rules of CivilProcedurePermit?8WILLIAMANDMARYBILLOFRIGHTSJ.199(1999).

22

theremainingclaims.Assumingthesamplegroupisrepresentativeoftheclassasawhole,

extrapolationerrorisafunctionofclaimvariability.Ifallclaimswerehomogeneous,

deservingequalawardsofdamages,thenapplyingtheaveragedeterminationinthesample

groupoverallclaimsintheextrapolationgroupwouldcauseminimalerror.Ontheother

hand,iftheclassentailsahighdegreeofclaimvariability,thenapplyingasinglepoint

estimate—suchastheaveragedeterminationinthesamplegroup—toallclaimsinthe

extrapolationgroupwillresultinahighdegreeoferror.Thisisalegitimateconcern;and

individualizedlitigationisaneffectivemeansofaddressingsucherror.Thatis,regardless

thedegreeofclaimvariability,ifclaimsaredecidedindividually,thereisnoextrapolation,

andthereforenoextrapolationerror.

However,opponentsofsamplinghaveneglectedadifferentformofvariability:even

iftheclaimsinaclassare(i)entirelyhomogeneous,and(ii)litigatedindividually,there

wouldneverthelessbesignificantvariabilityintheverdictsdeterminedforthoseclaims.35I

refertothisformofvariabilityasjudgmentvariability.Judgmentvariabilityresultsfroma

rangeofsources,includingjurycomposition,variabilityinthepresentationofevidence,

variabilityinthejudgeassignedtopresideoverthecase,etc.

Judgmentvariabilitycanbeunderstoodasaverysignificantsourceoferrorinthe

law.Forexample,imagineaclassof1millionhomogeneousclaimsbroughtinthecontext

ofaclassactionlawsuit(against,forexample,thelargestcigarettemanufacturers).Ajury

35See Saks&Blanck, supranote2;EdwardK. Cheng,When10Trials areBetterThan1000:AnEvidentiaryPerspectiveonTrialSampling,160U.OFPENN.L.REV.955,958-60(2012).

23

verdictintherepresentativeactionthatismerely$1,000greaterorsmallerthanthe

correctverdict—adifferentialcausedbyjudgmentvariability—wouldresultinawhopping

totalerrorvalueof1,000,000∗$1,000=$1billion.

Courtsareinfactwillingtoacceptadegreeoferrorinordertoavoidtheburdens

associatedwithlitigating1millionclaimsindividually.Buttheerrorthatcourtshave

acknowledgedandacceptedinreturnforsuchproceduralefficiencyisthatassociatedwith

variabilityinclaims,ratherthanthatassociatedwithjudgmentvariability.36Indeed,even

litigatingeachofthemillionclaimsindividuallywouldresultinaverylargeerrorvalue,as

eachofthemillionclaimsinvolvesanerrortermresultingfromjudgmentvariability—and

sothetotalerrorstillinvolvestheaggregationof1millionerrorterms.

Butimaginenowa(costly)hypotheticalprocedureinwhicheachandeveryclaim

werelitigated10timesindependently,andinwhichtheoutcomeassociatedwitheach

claimwerecomputedbytakingtheaverageofthe10verdictsassociatedwiththatclaim.

Thatis,startbytakingthefirstclaimandlitigateitbeforetenindependentjuries37to

obtain10independentverdicts.38Then,assigntheaverageofthe10verdictsasthe

36Ofcourse,iftheclaimsaretooheterogeneous,thecourtwillrefusetocertifytheclass.Theclasscertificationproceduredescribedabove is intendedtoensuretherepresentativenessof the litigatedclaims—inessencesomesignificantdegreeofhomogeneity.Tobeclear,ourcurrentexampleinvolveshomogeneousclaimsandthereforezeroclaimvariability.Thecourtsarethereforelikelytocertifysuchaclass.

37Letusfornowignorecomplexitiesassociatedwithstatisticalindependence.

38Wecanimagineasingletrialbefore10independentjuries;butitisbettertoimaginethecasebeinglitigatedanewbeforeeachofthe10 juries,since judgmentvariabilityarises fromvariability in litigation(e.g., inthepresentationofevidence)aswellasjuryvariability.

24

outcomeofthefirstcase.Byapplyingthisaggregatedoutcome,ratherthananysingle

verdict,wemayreducetheerrorresultingfromjudgmentvariabilitytonearlynothing.39

Applyingthisreplicationproceduretoallmillionclaimswouldbeincredibly

costly—perhapsfarmorecostlythancanbejustifiedbythebenefitsoftheprocedure—but

thepurposeofthehypotheticalissimplytomakethepointthatreplicationallowsusto

reduceerrorassociatedwithjudgmentvariability.

Now,inthecontextofasingleclaim,thebenefitsofthecostlyreplicationprocedure

arerelativelylow,sincetheerrorresultingfromjudgmentvariabilityislowinthefirst

place,asitistheerrorassociatedwithonlyasingleclaim.Similarly,applyingreplicationto

eachandeveryclaiminthehypotheticalsetof1millionclaimsdescribedaboveis

incrediblycostlyandimpractical;andinfactcanbeviewedjustasweviewreplicationina

singleclaim,but1milliontimes.

Amazingly,however,inthecontextofahomogeneousclass,wedonotneedtoapply

replicationtoeachandeveryclaiminordertorealizethebenefitsofsuchreplication.In

fact,assumingthattheclaimsarehomogeneous,wecanapplythereplicationprocedurea

singletime—eitherlitigatingasingleclaim,e.g.,before10independentjuries,orby

litigating10claims,eachbeforeanindependentjury40—andrealizetheerror-reducing

benefitsoftheprocedureasthoughweappliedreplicationforeachclaimindividually!

39Seesupranote28,andaccompanyingtext.

40Theseproceduresareequivalent,sincetheclaimsarehomogeneous.

25

Forthisreason,theclassactioncontextmayprovideauniqueopportunitytoreduce

errorresultingfromjudgmentvariabilitywholesale—thatis,withoutincurringthe“full”

costsofsuchprocedures.Thus,ifreplicationcanbeusedtomitigatetheerrorresulting

fromjudgmentvariability,itispossiblethataggregation—andsamplingproceduresin

particular—canresultinfarmoreaccuracythaneventhepurportedidealofindividualized

litigation.41

Opponentsofsamplinghavelargelyneglectedormisunderstoodthepossibilitythat

samplingcanreducejudgmentvariabilityandtherebyincreaseaccuracyinthelaw.For

example,ProfessorRobertBone,commentingonSaksandBlanck′sreasoningregarding

theutilityofsamplingasafunctionofclaimvariability,42statesthatthe“problemwith

[their]argumentiseasytosee,”remarkingthatifclaimswerehomogeneous,thensampling

wouldbeunnecessaryinthefirstinstance.43Theauthorexplainsthat“[i]ftherewere

perfecthomogeneity,therewouldbenoproblemwithmasstortlitigationfromthepointof

viewofoutcomeaccuracy.Ifallcaseswereidentical,thenthetrialofanyonewouldbejust

asgoodasthetrialofanyother,andtheaggregationcouldbeeasilyadjudicatedwitha

41Note,dependingontheseverityofthejudgmentvariabilityassociatedwithagivensetofclaims,aggregationmaywellenhanceaccuracyevenifreplicationyieldsabiasedoutcome.Thatis,ifwerelaxtheassumptionin§3.1above,thataveragingverdictsfrominfinitelymanyrepeatedtrialswouldyieldthe“correct”outcome,orifweotherwiseassumethataggregationwithrespecttoaparticularsetofclaimsislikelytoresultinabiasedoutcome—an outcome that does not approach the “correct” outcome as we average more and moreindependentverdicts—thenaggregationmayneverthelessenhanceaccuracyifthereductioninerrorresultingfromjudgmentvariabilityexceedstheerrorresultingfromthebias.

42SeeSaks&Blanck,supranote2,at833-37.

43RobertG.Bone,StatisticalAdjudication:Rights,Justice,andUtilityinaWorldofProcessScarcity,46VanderbiltL.Rev.561,578(1993).

26

simpleclassaction.”44ButProfessorBone′sreasoningneglectstoaccountfortheerror

associatedwithjudgmentvariability.Becausejudgmentvariabilityexistsirrespectiveof

thedegreeofclaimvariability,theerrorresultingfromasimpleclassactionmaybefar

greaterthanthatresultingfromasamplingbasedaggregationapproach.

Asdiscussedfurtherbelow,homogeneityprovidesidealcircumstancesforerror-

reducingsamplingandextrapolationprocedures.Further,itisimportanttounderstand

that,withrespecttooutcomeaccuracy,homogeneitydoesnotallowforeasyadjudication

“withasimpleclassaction,”asProfessorBonesuggests.Tothecontrary,asingleclass

adjudicationforalarge,evenhomogeneous,classwillresultinajudgment-errorvaluethat

ismultipliedovertheentireclassofclaims,asillustratedinthehypotheticalabove

involving1millionclaimsagainstthelargecigarettemanufacturers.

Ofcourse,evenclaimsintheclassactioncontextarenotperfectlyhomogeneous.

Buttheerror-reducingbenefitsofsamplingdonotnecessarilyrequirehomogeneity.

Indeed,asshownbelow,whetherclaimaggregationincreasesaccuracydependson

whethertheerrorresultingfromjudgmentvariabilitydominatestheerrorassociatedwith

claimvariabilityorviceversa.45

4. AFrameworkforExaminingSamplingandAccuracyinClassAction

Litigation

44Id.at578n.48.

45SeeSaks&Blanck,supranote2,at833-37.

27

Assuggestedintheprevioussection,replicationcanserveasafundamentaltoolfor

addressingjudgmentvariability;andtheclassactioncontextmayoftenallowcourtsto

realizethebenefitsofreplicationprocedureswithoutincurringtheenormouscostssuch

proceduresnormallyentail.InCimino,forexample,thecourtsamplednumerousclaims

withineachdiseasecategorytoarriveatanaggregateddetermination,whichwouldbe

appliedtotheunlitigatedclaimswithineachcategory,respectively.Efficiencywasthe

primarypurposeoftheprocedure;butgivenacertaindegreeofhomogeneitywithineach

diseasecategory,theaggregateddeterminationshadthepotentialtobemoreaccuratethan

theindividualizeddeterminations.

Importantly,thecourtinCimino,andothercourtsemployingaggregation

procedures,areconstrainedwithrespecttotheprocedurestheymayapply.HadJudge

Parker,inCimino,believedwithconfidencethattheaggregateddeterminationsweremore

accuratethantheindividualizeddeterminations,andhadhedesignedthetrialplanwith

thesolegoalofmaximizingaccuracy—unconstrainedbyotherconsiderations—hemay

havedecidednotonlytoapplytheaggregateddeterminationstotheunlitigatedclaims,but

toapplythemtothelitigatedclaimsaswell.Thatis,basedontheassumptionthatthe

aggregateddeterminationsaremoreaccuratethanindividualizeddeterminations,in

additiontoapplyingtheaggregateddeterminationinagivendiseasecategorytothe

unlitigatedclaimsinthatcategory,JudgeParkermayhavedecidedtoreplacethe

individualizeddeterminationsinthesamplegroupofthatcategorywiththecategory′s

aggregateddetermination-theaveragedamagesawardinthatcategory.Otherwise,onlythe

28

extrapolationgroupwouldreceivetheaccuracy-enhancingbenefitsofreplication,while

thesamplegroupwouldsufferahighdegreeofjudgmentvariability.

Thus,ifacourtpresidingoveranactioninvolvingahomogeneousclasswere

unconstrainedinitsdeterminationtomaximizeaccuracy,itmay,forexample,sampleall

claimsintheclassforindividualizedlitigationandthenreplaceallindividualized

determinationsbyanaggregateddetermination.

Infact,acourtmaycontinuetoreducejudgmentvariabilitybyreplicatingbeyond

thenumberofclaimsintheclass—forexample,bylitigatingeachclaimintheclasstwice.

Replicationiscostly,however,andpresumablythecourt′sdesireforaccuracywouldbe

constrainedbylitigationcosts.

Butmoreover,courtspresidingoverclassactionsareconstrainedbylegal

considerationsextraneoustoconcernsforaccuracyorefficiency.

Inthecurrentarticle,Iassumethatacourt′sabilitytoincreaseaccuracyis

constrainedinaparticularmanner:thatis,Iassumethatacourtwillnotreplacean

individualizeddeterminationwithanaggregateddetermination.Acourtmustchoose

whethertosampleagivenclaimforindividualizedlitigationortopreserveitseligibilityto

receiveanaggregateddeterminationextrapolatedfromthesampledclaims.Thus,ifacourt

samplesaclaimforindividualizedlitigationandthedeterminationofanindividualized

verdict,itwillnotlaterreplacetheindividualizeddeterminationwithanaggregated

determination,suchasthesample-groupaverage.

29

Itisappropriatetoimposesuchaconstraint,evenassumingacourt′swillingnessto

permitsampling-basedaggregation.Whilethereisclearprecedentfortheimplementation

ofproceduresinwhicharepresentativeclaimislitigatedanditsdeterminationextendedto

otherclaims,46thereisgenerallynoprecedent(withinthecontextofsamplingorother)for

aprocedureinvolvingthereplacementofanindividualizeddeterminationwithan

aggregateddetermination.

Indeed,thereisafundamentaldistinctionbetweenextendingrepresentative

determinationstounlitigatedclaimsandreplacingindividualizedverdictswithaggregated

verdicts.Forexample,althoughsomecourtshaveheldthatadefendant′sSeventh

Amendmentrighttoajurytrialisnotviolatedbytheformer,47aprocedureinwhicha

courtreplacesindividualizedjuryverdictswithaggregatedverdictsishighlylikelytobe

violativeoftheSeventhAmendment.48AsJusticeBrandeisexplained:

[TheSeventhAmendment]doesnotprohibittheintroductionofnewmethodsfor determining what facts are actually in issue, nor does it prohibit theintroductionofnewrulesofevidence.Changes in thesemaybemade.Newdevicesmaybeusedtoadapttheancientinstitutiontopresentneedsandtomakeofitanefficientinstrumentintheadministrationofjustice.…Indeed,such changes are essential to the preservation of the right. The limitation

46 In addition to sampling procedures, the class action device itself arguably provides for a form of suchtreatment.

47See,e.g.,InreEstateofMarcosHumanRightsLitigation,910F.Supp.1460,1468-69(D.Hawai′i1995)(“Here,thejurydiddeterminethefactsofthecase,asthesubstanceoftheactionwaspresentedtothejury.Therewouldbenobenefittoeithersideinhavingtheentireclasstestifygiventherepetitionintheclaims.Rule23oftheFederalRule[s]ofCivilProceduredoesnotmandatethepresenceofeachmemberoftheclass.Therefore,bychoosingarandomsampleof137claimantsinanaggregatetrial,neithersidewasdeprivedofeventheformoftheirrighttoajurytrial.”).

48SeeParklaneHosieryCo.v.Shore,439U.S.322,345-46(1979)(Rehnquist,J.,dissenting)(cautioningagainstallowingproceduraldevicesthatinvadethe“jury′sprovince,”itsfunctionasafact-finder);BlueCrossandBlueShieldofNewJersey,Inc.v.PhilipMorris,Inc.,178F.Supp.2d198,256(E.D.N.Y.2001).

30

imposedbytheamendmentismerelythatenjoymentoftherightoftrialbyjurybenotobstructed,andthattheultimatedeterminationofissuesoffactbythejurybenotinterferedwith.49

Whereasthereisprecedentforextendingrepresentativeverdictstoaclassof

claims,replacingindividualizedverdictswithaggregateddeterminationsarguably

substitutesthejury′spowertodetermineissuesoffactwithadeterminationarrivedatby

thecourt′saggregationprocedure.

Similarreasoningmayapplywithrespecttodueprocessconcernsaswell.Even

JudgeParker—who,notwithstandingultimatereversalbytheFifthCircuit,stronglysided

withthelegalityofsamplingproceduresinCimino—implementedaplanthathighlighted

theimportanceofallocatingindividualizedawardstothesampledclaimants,inadditionto

ensuring“representativeness”ofthesampledclaims.Arguably,thesefeaturesofthetrial

planwerecriticaltodistinguishingthesamplingprocedureinCiminofromthe“lumpsum”

approachrejectedbytheFifthCircuitinInreFibreboardCorp.50

Further,problemswithverdictreplacementareexacerbatedbythefactthatacourt

willnotknowwithcertaintywhethertoattributeobservedoutcomevariabilityto

judgmentvariabilityortoclaimvariability,andthereisnoprecedentforattributing

observedoutcomevariabilitytotheformerratherthanthelatter.Indeed,dueprocess,the

SeventhAmendmentrighttojurydetermination,andcurrentnotionsofproceduraljustice

49BlueCrossandBlueShieldofNewJersey,Inc.v.PhilipMorrisUSAInc.,344F.3d211,225-26(2dCir.2003)(quotingInrePeterson,253U.S.300,309-10(1920)).

50893F.2d706(5thCir.1990).SeeCimino,751F.Supp.at664-66;Saks&Blanck,supranote2,at823-24.Thus,iftheprocedureinCiminohadnotultimatelybeenrejectedbytheFifthCircuit,thecourtwouldhaverequiredindividualizedallocationofdamageawardstosampledclaimantsnevertheless.

31

likelyrequirecourtstoattributevariabilityinverdictstothelatter,andsimilarly,likely

precludethepossibilityofreplacinganindividualizeddeterminationwithanaggregated

one-evenassumingcourts′willingnesstosampleandapplyanaggregateddeterminationto

unsampledclaims.Forexample,ifajuryawards100distinctdamageawardsto100

sampledclaimants,theSeventhAmendmentlikelyprecludestheassumptionthatsuch

variabilityisduetojudgmentvariability,ratherthantothejury′sfindingofdifferences

amongtheclaims,anditlikelyforbidsthereplacementofsuchawardswithanaggregated

determination.51

Asasidenote,rememberthatthedrivingforcebehindargumentsforaggregation

hasbeenefficiencyandameansofaddressing“challengesofthemassrepetitivewrong.”In

additiontotheargumentsabove,theconstraintservesasanaturalboundaryonacourt′s

51 See generally Blue Cross and Blue Shield of New Jersey, 178 F. Supp. 2d at 256 (“The historical recorddemonstratesthattheFramer′smainobjectiveindraftingtheSeventhAmendmentwastolimittheabilityofanappellatecourttooverturnaciviljury′sfindingoffact.Thereisnoindicationtheyintendedtoconstrainthetrialjudge′ssubstantialdiscretiontoemployappropriateproceduralmechanismsinmanagingatrialsoastoarrive at the truth—or as near to the truth as time and humankind′s limitations allow.” (citing Simon v.PhilipMorris,Inc.,200F.R.D.21,33(E.D.N.Y.2001))).Notethatcourtsthathaveapprovedsampling-basedtrialplans have consistently permitted extrapolation only with respect to the unlitigated claims—those in theextrapolationgroup—while assigning the respective individualizeddeterminations to claims in the samplegroup.See,e.g.,Hilao,103F.3dat782-87;Cimino,751F.Supp.at652-54.Itisoftenunclearwhethercourtshavedonesoduetothepossibilitythattheyhavetreatedsamplingasasecond-bestoption,tobeappliedonlywhen individualized litigation is not practicable, or because they have presumed that replacing anindividualized verdict with an aggregated determination is precluded by the Constitution and rules ofprocedure. But numerous courts and scholars have emphasized the advantages of aggregation overindividualizedlitigationinthecomplex-litigationcontext.SeeCimino,751F.Supp.at659-66.SeegenerallyBlueCrossandBlueShieldofNewJersey,Inc.,178F.Supp.2dat248,258(findingthat“[w]hen,asinthecaseatbar,theplaintiff is an entity that has suffered its injury in the aggregate, statistical evidence is notably amoreaccurate and comprehensible form of evidence thanwould be the testimony ofmillions of smokers”; andemphasizingthat“[m]anyauthoritiesdemonstratethatjuryfactfindingisenhancedbytheuseofaggregatingtechniques” and that “[t]he essential functions of the Seventh Amendment are enhanced, not limited, byprocedureswhich streamline and focus jury fact-finding.”) Further, for the reasons above, it is likely thatregardlesswhether courts view sampling as a second-best strategy, theywouldbeunwilling to replace anindividualizedoutcomewithanaggregatedone.

32

ambitionstoreachhigherandhigherdegreesofaccuracy.Theconstraintimpliesthe

existenceofanoptimalsamplesize,withrespecttoaccuracy,asopposedtoconditions

underwhichacourtcancontinuetoincreaseaccuracybyincreasingreplication.

Thus,tointroduceaframeworkforanalyzingthecircumstancesunderwhich

aggregationcanincreasetheaccuracyoflegaloutcomes,Iwillassumeconditionsthatcan

bedescribedas“reductivesampling.”Toillustrate,supposewehaveapondofsickfish,and

ourgoalistosaveasmanyfishaspossible.Weneedto“sample”fish,andtherebyperform

testsonthem,inordertodeterminethenatureofthediseaseandhowtotreatit.Now

assume(i)thatasweincreasethenumberoffishwesample,weincreasetheaccuracyof

ourinformationregardingthenatureofthediseaseandhowtotreatit;but(ii)that

“sampling”afishrequiresthatwekillitinordertoperformtestsonit,etc.Thus,weare

presentedwithatradeoff:themorefishwesample,themoreaccurateourinformation

regardingthediseaseandhowtotreatit,butsincesamplingafishinvolveskillingit,the

morefishwesample,thefewerfishremaintotreat.Ifwesamplenofish,theywillall

eventuallycontractanddieofthedisease.Ontheotherhand,ifwesampleallofthefishin

thepond,wewillhaveahighdegreeofknowledgeofthediseaseandhowtotreatit,butwe

willhavenofishremainingtotreat!Thus,assumingcircumstancesdescribedbyreductive

sampling,inwhichsamplingaunitreduces,or(ashere)altogetherdestroys,thevalue

obtainedfromlaterextrapolationwithrespecttothatunit—forexample,treatingor

otherwiseapplyinganoperationtotheunit—leadstotheemergenceofanoptimization

33

problem:howmanyfishmustwesamplesoastobalanceourinterestsinlearninghowto

treatthediseaseontheonehandandpreservingfishtotreatontheother.

Bysampling,wepartitionthepopulationofclaimsinto(i)asamplegroup,inwhich

claimsarelitigatedandverdictsdeterminedonanindividualizedbasis;and(ii)an

extrapolationgroup,inwhichclaimsreceivetheaverageofthedeterminationsinthe

samplegroup.Themodelthereforehighlightsthetradeoffbetweentheaccuracyofthe

samplemean,ontheonehand,andthenumberofclaimsthatremainintheextrapolation

group,ontheother.

Inearliersections,Ihavehighlightedtwoformsofvariability:claim(i.e.,fact)

variabilityandjudgmentvariability.Claimvariabilitymightinclude,forexample,various

diseasetypessufferedbysmokersinaclasssuingacigarettemanufacturer.Judgment

variability,ontheotherhand,referstothevariabilityintheverdicts,holdingthefacts

acrossclaimsconstant.Asmentionedabove,judgmentvariabilitymightresultfrom

variabilityinthepresentationofevidence,variabilityinthecompositionofajuryand

variabilityintheselectionofthejudgetopresideoverthecase.

Letusbeginbyassumingthatjudgmentvariabilityisthesolesourceofvariability.

Thatis,assumethatclaimsareperfectlyhomogeneous.Thisisagoodstartingpointfor

analysisfortworeasons.First,wemayinfactobserveasignificantdegreeofclaim

homogeneityinthecontextofclassactionlawsuits.Moreover,aclassmaybesubclassified

forthepurposeofcreatingahighdegreeofhomogeneitywithineachsubclass.Second,

assumingzerofactvariabilityallowsexaminationofthefulltheoreticalbenefitof

34

aggregationbeforeaddressingtheproblemofbalancingconcernsofjudgmentvariability

withthoseoffactvariability.

4.1 HomogeneousClaims

AssumewehaveaclassofNhomogeneousclaimsfromwhichwesamplenclaims.

Letµbethetruemeanoftheverdictsoverthetotalpopulationofpossibleverdicts—i.e.,

the“correct”verdict.Let𝑋"bearandomvariabledefinedbytheactualverdictinthei-th

claim,fori=1,2,3,...,n,where𝑋"~𝑁(𝜇, 𝜎)),i.i.d.Further,define𝑋+asthemeanofthe𝑋" ,

fori=1,2,3,...,n,where𝑋+~𝑁(𝜇,,-

+).52

Then,pursuanttotheabovedescription,letusrepresentthetotalerrorassociated

withallNclaimsasthefollowingsumofsquaresresidual:

𝑅 = (𝑋" − 𝜇))+

"23

+ (𝑁 − 𝑛) 𝑋+ − 𝜇 )

Note,ifallclaimsarelitigatedindividually,thenn=N,andthisexpressionsimplifies

to:

𝑅"+6"7 = (𝑋" − 𝜇))8

"23

52Note, in actuality, it is unlikely that𝑋" has a normal distribution. For example, there is a relatively highprobabilityofadamagesvalueofzero,sincethisvalueisassociatedwiththeplaintiffs′ failuretoprovethedefendant′s liability. Amixture distributionmay bemore appropriate. Regardless, let us assume a normaldistributionforpurposesofsimplicity.

35

Letusnowsolvefortheoptimalsamplesize—thatis,thesamplesizen*that

minimizestheexpectationof𝑅,andthusmaximizes,onaverage,theaccuracyofthetotal

legaloutcome.

TheexpectationofRcanbederivedasfollows:

𝐸 𝑅 = 𝐸 (𝑋" − 𝜇))+

"23

+ 𝑁 − 𝑛 𝑋+ − 𝜇 )

= 𝑛𝜎) + 𝑁 − 𝑛𝜎)

𝑛

≡ 𝑓 𝑛

Now,tofindtheminimizingcondition,letussolve6<(+)6+

= 0.

min+𝑓 𝑛 ⇔

𝑑𝑓(𝑛)𝑑𝑛 =

𝑑𝑑𝑛 𝑛𝜎) + 𝑁 − 𝑛

𝜎)

𝑛 = 0

⇔ 𝑛 = 𝑁

Thus,onaverage,weminimize𝑅,andmaximizetheaccuracyoflegaloutcomes,by

setting𝑛∗ = 𝑁.Therefore,wemaximizeaccuracynotbylitigatingeachoftheclaims

individually,butratherbysamplingandindividuallylitigating𝑛∗ = 𝑁claims,and

applyingthesamplemean𝑋+∗asthelegaloutcomeinallclaimsintheextrapolationgroup.

Letusnowrelaxtheassumptionofhomogeneousclaimsinordertoexaminethe

constraintthatclaimvariabilityplacesonthebenefitsofsampling.

4.2 HeterogeneousClaims

Letusbeginbynotingthattheanalysisaboveallowsustoidentifyboundsonthe

optimalsamplingofclaimswithrespecttoclaimvariability.Specifically,under

36

circumstancesofzeroclaimvariability,ourresultundertheassumptionofhomogeneous

claimsapplies:wemaximizetheaccuracyoflegaloutcomesbysampling 𝑁claims.Onthe

otherhand,ifclaimvariabilityislarge,thenouroptimalsamplesizeisN.Thus,under

circumstancesofheterogeneousclaims,theoptimalsamplesizewillfallbetween 𝑁and

N.

Letusdevelopahierarchicalmodeltodeterminewherewithintheseboundsthe

optimalsamplesizefalls.Asbefore,assumewehaveaclassofNclaimsfromwhichwe

samplenclaims.Now,however,let𝜇" bethetruemeanoftheverdictsoverthetotal

populationofpossibleverdictsinthei-thclaim—i.e.,the“correct”verdictinthei-thclaim.

Thus,asopposedtocircumstancesofclaimhomogeneity,whereasingleµwasusedto

representthecorrectverdictinallNclaims,herewemustdefineasetofterms,

𝜇3, 𝜇), … , 𝜇8,torepresentthetruemean—the“correct”verdict—foreachoftheNclaims.

Let𝑋" bearandomvariabledefinedbytheactualverdictinthei-thclaim,fori=1,2,

3,...,n,where𝑋"~𝑁(𝜇", 𝜎)),independent.Let𝜇"bearandomvariabledefinedasthetrue

meaninthei-thclaim,where𝜇"~𝑁(𝜇, 𝜏)),i.i.d.Anddefine𝑋+asthemeanofthe𝑋" ,fori=

1,2,3,...,n,where𝑋+~𝑁(𝜇, 𝜎) + 𝜏)).

Then,pursuanttotheabovedescription,letusrepresentthetotalmeasurement

errorassociatedwithallNclaimswiththefollowinglossfunction:

𝑅 = (𝑋" − 𝜇"))+

"23

+ 𝑋+ − 𝜇F)

8

F2+G3

37

Itisimportanttoobtainagoodestimateofthedistributionofthe𝜇" ’s.Letusnow

assume,however,thatthe𝜇" ’sfollowanormaldistributionasdefinedabove.

Asbefore,wecanobtaintheoptimalsamplesizebysettingthefirstderivativeofthe

expectationoftheabovelossfunctiontozero.Wethusfindthattheoptimalsamplesize

undercircumstancesofheterogeneousclaimsandjudgmentvariabilityis

𝑛∗ = 𝑁1 + 𝜆)

1 − 𝜆)

where𝜆) = J-

,-.

Thisresultisintuitive.Considerthefollowingthreescenarios.First,ifthelevelof

claimheterogeneityiszero(τ2=0),then𝜆) = 0and𝑛∗ = 𝑁,theresultobtainedfromour

optimizationundercircumstancesofhomogeneousclaimsabove.Second,if𝜏) ≥ 𝜎),then

𝜆) ≥ 1and𝑛∗ = 𝑁.Third,if0 < 𝜆) < 1,and𝜏) < 𝜎),then𝑛∗ = 𝑁 3GM-

3NM-.

5. Discussion

Thecurrentsectionintroducesanumberofadditionaltopicsfordiscussionand

futureresearch.Specifically,Iintroducepossibilitiesforparameterestimation,sequential

sampling,stratificationandanumberofconsiderationsregardingincentivestoinitiate

claimsandtosettle.

5.1 ParameterEstimation

Thecurrentarticleisintendedtointroduceaframeworkforconsideringaccuracyin

thelaw,andinparticular,theconditionsunderwhichsampling-basedaggregationcan

increaseaccuracy.Itspurposeisnot,ingeneral,tomakenormativestatementsregarding

38

whetherorhowtoapplysamplinginagivencase.However,centraltoanypractical

implementationoftheframeworkintroducedhereinistheabilitytoestimatethejudgment

andclaimvariabilityparameters.Itisbeyondthescopeofthisarticletoaddressparameter

estimationindetail,butinthecurrentsection,Ihighlightanumberofimportant

considerations.

First,itisnotnecessarilythecasethatpracticalapplicationoftheframework

requiresapreciseestimationof,orevenassignmentofnumericalvaluesto,thevariability

parameters.Asshownabove,whenasetofclaimsishomogeneous(orapproximately

homogeneous),theoptimalsamplesizeisnotafunctionofvariability—itisthesquareroot

ofthenumberofclaims,regardlessthevalueofjudgmentvariability.Further,evenin

circumstancesinwhichtheclaimsarenotapproximatelyhomogeneous,theimportant

measurementistheratioofclaimvariabilitytojudgmentvariability—thatis,therelative

valueofclaimvariabilitywithrespecttojudgmentvariability.

Thus,asafirstcourseofaction,acourtcanattempttoidentifythesourcesofclaim

variabilityintheclassofclaims,andcan“reduce”suchvariabilitythroughstratification—

thatis,bybreakingtheclassintomultiplesubclasses.AsinCimino,discussedabove,acourt

maybeabletocreatemultiplerelativelyhomogeneoussubclassesratherthanasingle

heterogeneousclass.Idiscussstratificationfurtherbelow.

Atsomepointofstratification,orperhapsevenwithoutstratification,acourtmay

beableto“eyeball”therelativelevelofclaimvariability,andavoidtheneedforformal

estimationaltogether.Inasense,courtsalreadyanalyzethevariabilityofclaimsinthe

39

courseofthecertificationphaseoflitigation—forexample,intheiranalysisof

“commonality.”

Second,ifweareinterestedinfindingmoreaccurateestimatesofthemodel′s

variabilityparameters,wehaveanumberofoptions.Itispossiblethatwithinacertain

populationofindividuals(e.g.,individualsinagivengeographicregion),juryvariability—

likelyaverysignificantsourceofjudgmentvariability—issimilarfromcasetocase.In

otherwords,itmaynotbenecessarytostudyjudgmentvariabilitywithinanyparticular

classactionlawsuit.Instead,viewingjudgmentvariabilityasarandomvariable,itmaybe

possibletounderstandthedistributionofsuchvariabilitywithinagivengeographicregion

andcategoryofclaims.Thisisanimportantareaforfutureresearch.

Wecanofcourseobtainestimatesofjuryvariability,andevenofjudgment

variability,bylitigatingasingleclaimmultipletimes.Thiscanbedonewithinthecontextof

agivenclassaction,orsimplyinthecontextofasingleclaim,ormultiplesingleclaims

withinagivengeographicregion,inordertostudythedistributionofjudgmentvariability.

Asdiscussedabove,replicationiscostly;butitmaybedeemedworthwhileinlightofthe

potentialbenefitsofsampling.Further,judgmentvariabilitycanbeestimatedusingeither

simulatedlegaltrialsorperhapsasetofexistingcases.

Estimatingclaimvariabilityisinsomesensemorecomplexthanestimating

judgmentvariability.Buttwopossibilitiescometomind.First,anareaforfutureresearch

isthepossibilityofcategorizingpastlegalclaims,andundertakingastatisticalexamination

40

ofclaimvariabilitywithinsuchcategories.Thisresearchcanbeusedtoformulateclaim-

variabilityestimatesforfutureclaims.

Second,andperhapsmoresignificantly,courtscanaddresstheissueofestimating

claimvariabilitythroughsequentialsampling,whichIaddressinthefollowingsubsection.

5.2 SequentialSampling

Thesamplingframeworkintroducedaboveassumesthatwedesignateasamplesize

priortobeginningthesamplingprocedure.Forexample,underconditionsofhomogeneous

claims,wedecideupfrontthatwewillinclude 𝑁claimsinthesamplegroup,and𝑁 − 𝑁

claimsintheextrapolationgroup.Underconditionsofheterogeneousclaims,weinclude

𝑛∗ = 𝑁 3GM-

3NM-inthesamplegroup,and𝑁 − 𝑛∗claimsintheextrapolationgroup.

Asequentialsamplingprocedure,ontheotherhand,allowsustobeginbytakinga

smallsample,andthentousethissampletoestimaterelevantparameterspriorto

determiningamoregeneralsamplesize.Infact,wearenotrestrictedtoatwo-tier

samplingapproach.Wecanrepeatedlytakesmallsamples,anditerativelyupdateour

estimates.

Inparticular,acourtcanapplysequentialsamplingasfollows:(i)estimate

judgmentvariabilitythrough,forexample,oneoftheproceduresdiscussedabove;(ii)

estimateaprobabilitydistributionfortheclaimvariabilityparameterandchooseasample

sizepursuanttotheestimateandthesequentialsamplingplan;(iii)updatetheestimate

pursuanttotheclaimvariabilitydatainthesample,and,ifnecessary,takeafurthersample

pursuanttothenewestimate;and(iv)repeatstepthree.

41

Byfixingajudgmentvariabilityestimateupfront,wecanestimateclaimvariability

inthefirstsamplesimplybycomputingthesamplevarianceoftheoutcomesandadjusting

forjudgmentvariability.

Theparticularsofconstructingasequentialsamplingschemearebeyondthescope

ofthisarticle.Butnotethatitisimportantthattheschemeisconstructedsoastomaintain

independenceamongtheclaimoutcomes,whichmayinvolvecertainadditionalcosts.

Further,sequentialsamplinginvolvessomecommitmenttosamplinginthefirst

instance.Acourt,however,canimplementasequential-samplingprocedureincombination

withmethodsdiscussedintheParameterEstimationsectionabove.Forexample,acourt

mayusestratificationorsimply“eyeballing”toestimatethattheclassisbelowacertain

levelofheterogeneity—andtochooseaninitialestimatefortheclaimvariability

parameter.Thenthecourtcanchooseasamplesizeaccordingly.

Forexample,acourtmaybegininstageonebyassumingthataclassis

approximatelyhomogeneous,andthereforechooseasamplegroupofsize𝑛OPQPRS+PTU∗ =

𝑁,anoptimalsamplesizethatdoesnotdependonvariabilityparameters.Oncetheclaims

inthesamplegrouphavebeenindividuallylitigated,thecourtmayobtainanestimateof

theclaimvariability.53Now,sincethecourthasobtainedestimatesofclaimvariabilityand

judgmentvariability,itcanrelaxtheassumptionofhomogeneousclaims,andcompute

53Assumingwecanobtainareasonableestimateof judgmentvariabilityasdiscussedabove, thecourtmayobtain an estimate of claim variability simply by computing the variability of sample group outcomes andsubtractingoutthevariabilityattributabletojudgmentvariability.

42

𝑛∗ = 𝑁 3GM-

3NM-,themore-generaloptimalsamplesize.Ifthiscomputationresultsina

samplesizethatexceedsthesizeofthestage-onesamplegroup,thenthecourtwillsample

anadditional𝑛∗ − 𝑛OPQPRS+PTU∗ claimsforlitigation.Finally,oncethecourthaslitigatedn*

claimsandreceivedindividualizeddeterminationsfortheseclaims,itwillassignthemean

ofthesedeterminations,𝑋+∗ ,toallremainingclaims—theclaimscontainedinthe

extrapolationgroup.

5.3 Stratification

Theutilityofsamplingisassociatedwiththelevelofhomogeneityoftheclass.Atthe

extremes,aclassofperfectlyhomogeneousclaimsallowsforanoptimalsamplesizeof

𝑛OPQPRS+PTU∗ = 𝑁,whilecircumstancesunderwhichclaimheterogeneityishighand

judgmentvariabilityislowaltogetherdestroythevalueofextrapolatingfromthesample

group—thatis,suchconditionsinvolveanoptimalsamplesizeofN.Thus,tosimplifythe

problemandmaximizetheeffectivenessofclaimaggregation,acourtcanstratifya

heterogeneousclassofclaims,therebycreatingmultiplesmaller,butrelatively

homogeneous,classes.54Throughstratification,thecourtcan“control”forsignificant

sourcesofvariability,andthereby“modify”thefunctionalvariabilityoftheclaims,and

achievealevelofefficiencyandaccuracythatisotherwisenotpossible.

Asasimpleexample,assumethat5,000smokerclaimscanbedistinguishedfrom

oneanotheronlyonthebasisofthetypeofdamagessuffered.One-fifthoftheplaintiffs

54SeegenerallyCimino,751F.Supp.at649;Saks&Blanck,supranote2,at842.

43

sufferedfromheartdisease,one-fifthsufferedfromlungdisease,one-fifthsufferedmerely

fromoverpaying,etc.Assume,forsimplicity,nooverlapamongthecategories.Ratherthan

samplingfromtheentireclassof5,000claims𝑛∗ = 𝑁 3GM-

3NM-claims—arelativelylarge

samplesize,ifnotthesizeoftheentire5,000-claimclassaltogether—thecourtcancreate

five1,000-claimhomogeneousclasses,fromwhichitwillsample𝑛"∗ = 𝑁claimsfrom

each(whereiisanindexforsubclasses1…5).Thus,intotal,thecourtwillsample5( 𝑁)

claims;but,asinCimino,itwillusethe𝑛3∗ = 𝑁3claimstoextrapolatedeterminationsfor

thefirstsubclass,the𝑛)∗ = 𝑁)claimstoextrapolatedeterminationsforthesecond

subclass,andsoon.

Aninterestingtopicforfutureresearchistheproblemofoptimalstratification—i.e.,

thenumberofstrata—undervariousvariabilityconditions.

5.4 IncentivestoInitiateClaimsandtoSettle

Asmentionedabove,itisnotmyaiminthecurrentarticletomakenormative

determinationswithrespecttotheimplementationofclaim-aggregationprocedures.

However,animportantconsiderationforlawmakersisthewayinwhichsuchascheme

mightaffectincentivestoinitiateactions,aswellasincentivestosettleclaims(ratherthan

litigate),giventhatanactionhasbeeninitiated.

Atfirstglance,onemightthinkthatclaimaggregationmayhaveanegativeeffect,

withrespecttosocialobjectives,onincentivesofpotentiallitigantstoinitiateclaims.

Specifically,withinagivenclassofclaims,potentiallitigantswithweakerclaims(i.e.,claims

involvingalowerexpectedrecovery)maybeoverincentivizedtojoinaclassgovernedbya

44

procedureinvolvingaggregationofclaims,whilepotentiallitigantswithstrongerclaims

maybeunderincentivizedtojoinsuchaclass.Thisisbecauseclaimaggregationinvolvesa

substantialpossibilitythatagivenlitigantwillbechosenfortheextrapolationgroup,in

whichcase,shewillreceivethemeandeterminationinthesamplegroupratherthanan

individualizeddetermination.

Therearenumerousproblemswiththisreasoning;andtotheextentthatincentives

arenecessarilyaffected,courtsmaybeabletoaddresssucheffectsatarelativelylowcost.

First,totheextentthat“weaker”plaintiffsareoverincentivizedtojoinaclassunderthe

aggregationscheme,itisnotdissimilartotheoverincentiveproducedbythecurrentclass

actionframework,whereinaverdictassociatedwith“representative”claimsisappliedto

theentireclass.Ofcourse,acourtmayorderaparticularprocedure,suchasindividualized

hearings,fordeterminingtheallocationofthetotalawardamongtheclaimants;butsucha

procedureisentirelyconsistentwiththeaggregationschemeaswell.Forexample,acourt

mayemploysamplingforthepurposeofarrivingataclasswideliabilityordamages

determination,butthereaftermayimposeafurtherprocedurefordeterminingthe

allocationofthedamagesaward.

Underboththecurrentclassactionframeworkandasamplingframework,courts

mayavoidundesirableincentivesbydeterminingthataclassistooheterogeneous,thus

refusingtocertifytheclassforclassactionlitigation.Further,acourtmayseektostratify

theclassinthefirstinstanceandtherebycreaterelativelyhomogeneoussubclasses.Inthis

way,theweakerlitigantsarenotoverincentivized,andthestrongerlitigantsarenot

45

underincentivized,tojointheclass.Eachcategoryoflitigantswouldexpecttobegrouped

withothersimilarly-situatedlitigants.

Additionally,itiswellrecognizedthatsamplingoffersthepotentialforenormous

costsavings,andtotheextentthataggregationallowsforincreasedaccuracyand

reductionsinlitigationcosts,incentivestolitigateclaimstherebybecomemorealigned

withsocialobjectives.55Assertionsthatsamplingincreasessocialcostsbyburdeningthe

courtsystemwithnumerousbellwethertrialsareflawed.Suchassertionsarebasedonthe

assumptionthat,ifacourtrefusestocertifyaproposedclass,themembersoftheclasswill

treattherulingasagenerallossandwillnotbringtheirclaimsindividually.Totheextent

thatthisisinfactthecase(itoftenis),closingthecourt′sdoorsonlitigantswithvalid

claimscertainlyshouldnotbeunderstoodasasocialvictory.Tothecontrary,itrepresents

afailureofthejudicialsystem,andthesocialcostsofsuchfailures—including,forexample,

theharmfulincentivestructureestablishedbyallowingmasstortfeasorstoavoid

liability—arelikelytofaroutweighthelitigationcostsofsampling.56

55Thisapplieswhethersocialobjectivesrelate,forexample,toconcernsofdeterrenceorconcernsoffairness.

56SeegenerallyBlueCrossandBlueShieldofNewJersey,Inc.,178F.Supp.2dat247-48(“Inmassexposurecaseswithhundredsofthousandsormillionsofinjured...thecostofsuchone-on-oneproceduresisinsuperableandunsuitableforeitherajuryorabenchtrial.Theconsequenceofrequiringindividualprooffromeachsmokerwouldbetoallowdefendantswhohaveinjuredmillionsofpeopleandcausedbillionsofdollarsindamages,toescapeallliability.”);DavidRosenberg,TheCausalConnectioninMassExposureCases:A“PublicLaw”VisionoftheTortSystem,97HARV.L.REV.849(1984);DavidRosenberg,IndividualJusticeandCollectivizingRisk-BasedClaims inMass-ExposureCases, 71N.Y.U.L.REV. 210 (1996).Note, to the extent that itmaybe argued thatsamplingproceduresallowforincreasedaccuracyandshouldthereforeoccurevenwhensuchproceduresarenot required tomaintain a class action, sampling arguably results in heightened litigation costs—e.g., fromlitigating 𝑁 claimsrather thanasingleclaim—andwill thereforeaffect incentives to litigateandtosettle.Claimantsmaybelesswillingtobringclaims,andclaimantsanddefendantsmoreeagertosettlemattersoutofcourt,toavoidtherelativelyhighcostsoflitigation.Itmaybeargued,however,thatthecostsofjudgmentvariability,andtheinaccuracythatresultstherefrom,outweighanyincreaseinlitigationcost,includingalteredincentiveswithrespectto litigationandsettlement.That is,currentclassactionprocedures,whereasingletrial—subject to a high degree of judgment variability—can possibly determinewholesale the outcome of

46

Further,aggregationproceduresmaybedesignedtoencouragesettlementamong

theparties.Indeed,theuseofbellwethertrialsforpurposesoffacilitatingsettlementand

theresolutionofclaimsiswellrecognized.AstheFifthCircuitremarkedinInreChevron

U.S.A.,Inc.,“Thenotionthatthetrialofsomemembersofalargegroupofclaimantsmay

provideabasisforenhancingprospectsofsettlementorforresolvingcommonissuesor

claimsisasoundonethathasachievedgeneralacceptancebybothbenchandbar.”57

Forexample,acourtmayoptforaframeworkinwhichsampledclaimsarelitigated

instages.Thisnotonlyallowsforcalibrationofsamplingprocedures,asdescribedabove,

butalsoallowsthepartiestocalibratetheirexpectationswithrespecttotheoutcomeofthe

litigation,thusfacilitatingsettlement.TheFifthCircuitexplained,“Thereasonsfor

acceptancebybenchandbarareapparent.Ifarepresentativegroupofclaimantsaretried

toverdict,theresultsofsuchtrialscanbebeneficialforlitigantswhodesiretosettlesuch

claimsbyprovidinginformationonthevalueofthecasesasreflectedbythejury

verdicts.”58

thousands,orevenmillions,ofclaimsarguablyentailfargreatercoststhanthoseassociatedwithlitigatingtheclaims contained in the sample group. Future research should examine ways in which courts can takeadvantage of economies of scale to reduce litigation costs associated with sampling. Additionally, futureresearchshouldinvestigatetheeffectsonincentivestoinitiateclaimsandsettlesuitsafteraccountingfortheroleoflawfirmsandcontingencyfeesinparticular.Itispossiblethatsucheffectsaremitigatedafteraccountingforfirms′willingnesstoincurtheriskassociatedwithlawsuitsinexchangeforlargerpotentialpayoffs.Itisalsopossiblethatthecostsassociatedwithaggregationcombinedwiththeresultingincreasesinaccuracymayin fact result in a reduction in frivolous or otherwise undesirable litigation. In any event, it shouldnotbeassumed thata claimant in thesamplegroupwould incurcostsbeyond those incurredbyclaimants in theextrapolationgroup; such costs canbe spreadacross the class, if the class (rather than,e.g., a law firm) isincurringthecostsoflitigationinthefirstinstance.

57109F.3d at1019 (highlighting reference tobellwether trials in theManual forComplexLitigation, citingManualforComplexLitigation§33.27-.28(3ded.1995)).SeegenerallyFallonetal.,supranote12,at2323.

58InreChevronU.S.A.,Inc.,109F.3dat1019;seealsoConnecticutCoolingTotalAir,Inc.v.ConnecticutNaturalGasCorp.,46Conn.Supp.82,89(Conn.Super.Ct.1999)(“Thecourthasdevisedacasemanagementstrategy

47

Adefendantcorporationthatobservesconsistentlossesinthefirststageof

samplingmaydecidetobendtotheplaintiffs′settlementdemands.Similarly,aplaintiff

classthatobservesconsistentlossesmaydecidetodropfurtheractionaltogether.Between

thesetwoextremes,thepartiescancalibratetheirexpectationsandtheirdemands,and

therebyenhancethelikelihoodofsettlingafterperhapstheinitialstageoflitigation.

6. Conclusion

Courtsandauthorshavesuggestedthat,undercertaincircumstances,sampling

procedurescannotonlyincreaseefficiency,butaccuracyaswell.SaksandBlanck,aswell

asotherauthors,havehighlightedtheroleofwhatIhavereferredtoas“judgement

variability”inthelaw,andtheabilitytoemploysamplingprocedurestoreducesuch

variabilityandtherebyreduceerrorinthelaw.Suchassertionshavebeenusedtorebut

anti-aggregationargumentsthatarebasedonthepremisethataccuracycannotbe

sacrificedforthesakeofefficiency.

Butunderwhatcircumstancesinparticularcansamplingproceduresincrease

accuracy?Whataretheconditionsunderwhichsamplingwillresultinanetreduction—

accountingforeffectsofbothjudgmentvariabilityandclaimvariability—oferrorinthe

law?Assertionsthatsamplingprocedurescanincreaseaccuracyhavebeenmetwith

skepticismandageneralunwillingnesstorelyonsuchassertionsinreal-worldcontexts.

Suchskepticismisarguablyduetothefactthatlegalscholarshiphasnotbegunto

involvingtrialofa‘bellwether’casefromamongthesecases,asapotentialmeansofachievingamoreprompttrialofissuesthatmayleadtonegotiatedresolutionsofallthecases.”).SeegenerallyCimino,751F.Supp.at649.

48

contemplate,inanyrigorousform,thepracticaleffectofsamplingproceduresonaccuracy

underreal-worldconditionsofbothclaimandjudgmentvariability,andwithrealistic

constraintsimposedbythelaw.

Inthecurrentarticle,Ihaveintroducedaframeworkforexaminingtheconditions

underwhichsamplingcanincreaseaccuracyinthelaw.Inparticular,Ihaveintroduceda

simplemodelforstudyingtheeffectsofsamplingonaccuracy,andforderivingtheoptimal

samplesize,withrespecttoaccuracy,underconditionsofclaimandjudgmentvariability,

andwithconstraintsdescribedbyreductivesampling.Ihavealsointroducedanumberof

importantconsiderationsandtopicsforfutureresearch.Specifically,Ihavediscussed

possibilitiesforestimatingvariabilityparametersandtheuseofsequentialsampling,the

potentialforstratificationproceduresthatmayallowcourtstoreducethedegreeofclaim

heterogeneityandtherebyincreasetheutilityofclaimaggregation,andconsiderations

relatedtoincentivestoinitiateclaimsandtosettleundercircumstancesofclaim

aggregation.

Myaiminthecurrentarticlewastobuildonpreviousliteraturetointroducea

frameworkanddiscussionforconsideringtheconditionsunderwhichsamplingcan

increaseaccuracyinthelaw.Myhopeisthatthearticlewillserveasafirststeptowarda

morerigorousexaminationofsuchconditions,andultimatelythepracticalimplementation

ofsamplingtoincreaseaccuracy,aswellasefficiency,inthelaw.

49

II. SamplingandReliabilityinClassActionLitigation*

Courtscontinuetostrugglewiththelimitsofstatisticalsamplinginresolvingclaims

arisingfrom“themassrepetitivewrong.”1InCiminov.RaymarkIndus.,Inc.,a1990asbestos

classaction,U.S.DistrictCourtJudgeRobertParkerdividedaclassof2,298claimantsinto

fivesubclassesbasedonclaimedinjuries,andselectedarandomsamplefromeach

subclasstoformarepresentativesamplegroupof160claimants.2Hesubmittedthesample

group’sclaimstoajuryforindividualdeterminationsandthenextrapolatedoutcomesfor

thenon-sampledclaimsbyapplyingtheaveragesamplegroupdeterminationstoeach

subclass,respectively.3TheFifthCircuitultimatelyrejectedJudgeParker’ssampling

procedureonconstitutionalgrounds.4

Inrecentyears,courtsandcommentatorshavecriticizedtheuseofsamplingto

proveclasswideliabilityanddamages.5Itiswidelybelievedthatsamplingservesgoalsof

*Thisarticleoriginallyappearedin2016CARD.L.REV.DENOVO207(2016).Citation:HillelJ.Bavli,SamplingandReliabilityinClassActionLitigation,2016CARD.L.REV.DENOVO207(2016).Itmayhavebeenmodifiedfromitsoriginalformforpurposesofconsistency.TheauthorthanksJohnKennethFelterandtheCardozoLawReviewEditorialBoardfortheirhelpfulcommentsandedits.

1Ciminov.RaymarkIndus.,Inc.,751F.Supp.649,652(E.D.Tex.1990)(citingHERBERTB.NEWBERG,NEWBERGONCLASSACTIONS§17.06,at373(2ded.1985)).

2Id.at652–54;seealsoHillelJ.Bavli,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyinClassActionLitigation,14LAW,PROBABILITY&RISK67,68(2015)[hereinafterAggregatingforAccuracy].

3AggregatingforAccuracy,supranote2,at68.

4SeeCiminov.RaymarkIndus.,Inc.,151F.3d297(5thCir.1998).

5See,e.g.,Wal-MartStores,Inc.v.Dukes,131S.Ct.2541,2561(2011).

50

efficiency,butonlyatthecostofreliability—andthatthe“benefitsofefficiencycannever

bepurchasedatthecostoffairness.”6

Ina1992StanfordLawReviewarticle,ProfessorsMichaelSaksandPeterBlanck

arguedthatsampling,whenperformedcorrectly,notonlysatisfiesthestandardsof

reliabilityachievablethroughindividuallitigation,butthatsamplingcanalsoincreasethe

reliabilityoflegaloutcomes.7

However,courtshavegenerallybeenunwillingtoacceptthereliabilityofsampling

proceduresusedtoproveclasswideliabilityanddamages;andcourtsfrequentlyreject

samplingonconstitutionalandproceduralgrounds.8But,asSaksandBlanckhaveasserted,

“amajor—perhapsthemajor—dueprocessconcerninanaggregatedtrialisthevalidityof

theoutcome.”9Moreover,“[t]hemainargumentagainsttrialbyaggregationandsampling

assertsthatsuchtrialscannotgivethepartiesasaccuratearesultastheywouldobtain

throughtraditionalbilateraltrials.”10

6Malcolmv.Nat’lGypsumCo.,995F.2d346,350(2dCir.1993);seealsoInreChevronU.S.A.,Inc.,109F.3d1016,1023(5thCir.1997)(Jones,J.,concurring).

7MichaelJ.Saks&PeterDavidBlanck,JusticeImproved:TheUnrecognizedBenefitsofAggregationandSamplingintheTrialofMassTorts,44STAN.L.REV.815,815–19(1992).

8SeeWal-MartStores,Inc.,131S.Ct.at2561;seealsoOrtizv.FibreboardCorp.,527U.S.815,845(1999)(citingAmchemProd.,Inc.v.Windsor,521U.S.591,613(1997)).

9Saks&Blanck,supranote7,at833(emphasisinoriginal);seealsoJointAnti-FascistRefugeeComm.v.McGrath,341U.S.123,171–72(1951)(Frankfurter,J.,concurring);MartinH.Redish&LawrenceC.Marshall,AdjudicatoryIndependenceandtheValuesofProceduralDueProcess,95YALEL.J.455,476–77(1986).

10Saks&Blanck,supranote7,at833;seealsoInreChevronU.S.A.,Inc.,109F.3dat1020(“[O]urproceduraldueprocessconcernsfocusonthefactthattheprocedureembodiedinthedistrictcourt’strialplanisdevoidofsafeguardsdesignedtoensurethattheclaimsagainstChevronofthenon-representedplaintiffsastheyrelatetoliabilityorcausationaredeterminedinaproceedingthatisreasonablycalculatedtoreflecttheresultsthatwouldbeobtainedifthoseclaimswereactuallytried.”).

51

Thequestionremains:whyhavecourtsandcommentatorslargelydiscountedthe

argumentthatsamplingoffersnotonlyincreasedefficiency,butreliabilityaswell?Ina

2015paper,Iexplainthataprimarysourceofskepticismmaybethegeneralitywithwhich

theargumenthasbeenmade.11AlthoughSaksandBlanck,andothers,havesuggestedthat

samplingmayincreasethereliabilityoflegaloutcomes,theliteraturehasnotadequately

developedtheargumentorproducedaframeworktoanalyzetheeffectofsamplingon

reliability.12Indeed,samplingdoesnotinevitablyincreasethereliabilityoflegaloutcomes;

rather,itseffectonreliabilitydependsontheparticularfeaturesoftheclaims.13Therefore,

inAggregatingforAccuracy,Idevelopaformalframeworkforexaminingconditionsunder

whichsamplingcanincreasethereliabilityoflegaloutcomes.14

InthisArticle,IexplainmyconclusionsinAggregatingforAccuracyinnon-

mathematicalterms,andunderscorecertainimplicationswithrespecttoclassaction

litigationandconsiderationsinlightoftheU.S.SupremeCourt’srecentdecisioninTyson

Foods,Inc.v.Bouaphakeo.15Ibeginbydescribingthebuildingblocksofmyanalysis—the

conceptsofreliabilityandaccuracyinthelaw.

1. ReliabilityintheLaw

11AggregatingforAccuracy,supranote2,at69.

12Id.

13Id.

14Seegenerallyid.

15136S.Ct.1036(2016).

52

Assumethatforeverylegalclaimthereisa“correct”outcomethatcanbe

determinedbyapplyingthe“true”stateofthelawtothecompletefactssurroundingthe

claim.16But,intherealworld,completeknowledgeregardingthelawandthefacts

surroundingaclaimisunavailable.Consequently,alegaloutcome(resultingfromatrialor

otheradversarialproceeding)servesasanestimateofthe“correct”outcomefortheclaim.

Therefore,foreverylegalclaim,thereisanerrortermthat(althoughgenerallyunknown)

reflectsthedisparitybetweentheobservedoutcomeandthe“correct”outcome.Accuracy,

then,isdefinedintermsoftheproximityoftheobservedoutcometothe“correct”

outcome.17

Oneconvenientandarguablysensibleconceptualizationofthe“correct”outcome

associatedwithalegalclaimistheawardthatwouldresultfromcomputingthemeanof

infinitelymanyadjudicationsoftheclaimundervariousconditions(e.g.,variousjury

compositions,judges,attorneys,presentationsofevidence,etc.).18

Thus,Idefinethereliabilityofalegalprocedureastheaccuracyofthelegaloutcome

thatcanbeexpectedbyfollowingtheprocedure.Forexample,ifitisexpectedthatacertain

16Alternatively,itcanbeassumedthatthereisadistributionof“correct”outcomesassociatedwitheverylegalclaim.SeegenerallyAggregatingforAccuracy,supranote2,at69.

17Seegenerallyid.at74;JonathanJ.Koehler&DanielShaviro,VeridicalVerdicts:IncreasingVerdictAccuracyThroughtheUseofOvertlyProbabilisticEvidenceandMethods,75CORNELLL.REV.247(1990)(discussing“accuracy”).Notethataccuracycanbedefinedmoreformallyusingtheconceptsofbiasandvariance.However,anexplicitdiscussionoftheseconceptsisbeyondthescopeofthisArticle.

18SeeSaks&Blanck,supranote7,at833–34;AggregatingforAccuracy,supranote2,at74.Othermeasuresofcentraltendency,suchasthemedian,arealsopossible.Forsimplicity,throughoutthisArticleweusesolelythemean.

53

legalprocedurewillproduceahighlyaccurateoutcome—anobservedoutcomeinclose

proximitytothe“correct”outcome—theprocedureisconsideredreliable.19

Anin-depthdiscussionoftheroleofaccuracyinthelawisbeyondthescopeofthis

Article.Iassumethataccuracyisafundamentalgoalofthelaw,whethersuchgoalis

groundedinaccuracyitselforafurtheraim,suchasdeterrenceorfairness.20

2. ClaimandJudgmentVariability

Criticsofsamplinghavefocusedontheerrorthatresultsfromapplyingapoint

estimate—e.g.,theaverageofthesamplegroupoutcomes—toaclassofheterogeneous

legalclaims.21Heterogeneitycanbedescribedintermsofclaimvariability—i.e.,the

variabilityofthefactsoftheclaims,or,moreprecisely,thevariabilityofthe“correct”

outcomesassociatedwithindividualclaims.22

Butcriticsofsamplinghaveignoredasecondtypeoferror:errorresultingfrom

judgmentvariability.23Judgmentvariabilityrepresentstherandomnessassociatedwitha

claim’soutcome.Itarisesfromthevariabilityinconditionsunderwhichoutcomesare

determined,includingthecompositionofthejury,thejudgepresidingoverthecase,the

19SeeAggregatingforAccuracy,supranote2,at74–78foradetaileddiscussionofaccuracyandvariabilityinthelaw.

20Seeid.at75;Saks&Blanck,supranote7,at829(citingRedish&Marshall,supranote9,at476–77).

21AggregatingforAccuracy,supranote2,at75–76.SeegenerallySabyGhoshray,HijackedbyStatistics,RescuedbyWal-Martv.Dukes:ProbingCommonalityandDueProcessConcernsinModernClassActionLitigation,44LOY.U.CHI.L.J.467(2012).Iftherearesubclasses,thenacourtislikelytoassigntheaveragesamplegroupoutcomewithineachsubclasstothenon-sampledclaimsineachsubclass,respectively.See,e.g.,Ciminov.RaymarkIndus.,Inc.,751F.Supp.649,651–54(E.D.Tex.1990).

22AggregatingforAccuracy,supranote2,at76.

23Id.at76–78.

54

presentationofevidence,theattorneysinvolvedinthecase,etc.24Forexample,ifasingle

claimistriedtentimesindependently(eachtrialwithanewselectionoftrieroffact,

attorneys,etc.),itislikelythattherewouldbetendistinctverdicts.But,ifthereisasingle

correctoutcomeassociatedwiththeclaim,thenjudgmentvariabilityreflectserror—

disparitiesbetweentheobservedoutcomesandthe“correct”outcome.25

Forsimplicity,Iassumethat,onaverage,acasewillresultinthe“correct”

outcome—instatisticalterms,theoutcomeisunbiased.Judgmentvariabilityrepresentsthe

degreetowhichanobservedoutcomevariesaroundthe“correct”outcome.26Thereader

mayimagineabellcurvecenteredatthe“correct”outcome,wherethejudgmentvariability

determinesthewidthofthecurve.Onaverage,theoutcomewillbethe“correct”outcome;

buttheoutcomeisvariablearoundthe“correct”outcome,andsuchrandomness,

representedbyjudgmentvariability,reflectserror.27

Indeterminingappropriatestandardsintheclassactioncontext,courtsand

commentatorshavefocusedonerrorresultingfromclaimvariability,buthavegenerally

ignorederrorresultingfromjudgmentvariability.28Indeed,eventhepurportedidealof

individuallitigationproducesoutcomesthataresubjecttojudgmentvariability.

24Id.at76–77.

25SeeSaks&Blanck,supranote7;EdwardK.Cheng,When10TrialsAreBetterthan1000:AnEvidentiaryPerspectiveonTrialSampling,160U.PA.L.REV.955,958–60(2012);AggregatingforAccuracy,supranote2,at75–78.

26SeegenerallyAggregatingforAccuracy,supranote2,at75–78.

27Seeid.at75–79.Considerthemagnitudeoferrorthatresultsfromjudgmentvariabilitywhenclasswidedamagesaredeterminedbyadjudicatingasinglerepresentativeclaim.

28Id.at76–78.

55

3. ReducingJudgmentVariabilitywithSampling

Consideracostlyprocedurethroughwhichtheoutcomeofaclaimisdeterminedby

averagingtheverdictsresultingfromtenindependent“replications”ofatrial,or“repeated

adjudications”(involving,forexample,differentjudges,juries,attorneys,presentationsof

evidence,etc.).Assumingtheoutcomeisrelativelyunbiased,itiseasytoshowthat

followingthisprocedureresultsinanaccurateoutcome—anoutcomethatisclosetothe

“correct”outcome.Similarly,thisprocedurewillproduceanaccurateoutcomeforeach

claimofeachmemberofaputativeclass(orsubclass).Replicationthusincreasesthe

reliabilityoflegaloutcomesbyreducingtheerrorcausedbyjudgmentvariability.

Ontheotherhand,thecostsofsuchaprocedureareenormous,andlikelynot

justifiedbythebenefitsoftheprocedureforasingleclaim(orforeachputativeclass

member’sclaimthatrequiresapplicationoftheprocedurefortheindividualclaim).

Importantly,however,inaclassofhomogeneousclaims,itisnotnecessarytofollow

theprocedureforeachclassmember’sclaimtorealizethebenefitsoftheprocedure:

followingthereplicationprocedureforasingleclaimandextrapolatingtheresulttoall

claimsofthehomogeneousclass(orsubclass)willyieldthereliabilitybenefitsasthough

theprocedurewasfollowedforeachclaim.

Classactionsprovideopportunitiestorealizethereliabilitybenefitsofreplication

withoutincurringtheprohibitivecoststhatsuchproceduresentail.29Inparticular,courts

canusesamplingtoimprovelegaloutcomesbyreducingerrorresultingfromjudgment

29Id.at77.

56

variability.Moreover,samplingmayofferadegreeofreliabilitythatcannotbeobtained

eventhroughthepurportedidealofindividualadjudication.Further,although

homogeneityishelpful,itisnotnecessary.Asexplainedbelow,thebenefitsofsamplingina

heterogeneousclassdependonwhether,andtowhatextent,theerrorresultingfrom

judgmentvariabilitydominatestheerrorresultingfromclaimvariability.30

4. ConditionsUnderWhichSamplingCanImprovetheReliabilityofLegal

Outcomes

Above,Idescribehowrepeatedadjudicationcanincreasethereliabilityoflegal

outcomes.Assumingacertaindegreeofhomogeneity,ifacourtwereunconstrainedbycost

orlawinitspursuitofreliability,itmight“sample”alltheclaimsofaclassforindividual

adjudicationandthenreplaceallindividualoutcomeswithasingleaggregatedoutcome.In

fact,acourtcouldfurtherreducetheerrorresultingfromjudgmentvariabilityby

adjudicatingeachclaimtwice,ormoretimesforthatmatter.But,inadditiontothehigh

costsoflitigation,acourt’sabilitytoachieveextremelyreliableoutcomesisconstrainedby

law.

InAggregatingforAccuracy,Iarguethatacourtmaynotreplaceanindividualized

outcomewithanaggregatedone.31Ihighlightafundamentaldistinction,basedon

constitutionallawandrulesofcivilprocedure,betweenacourt’sauthoritytoextrapolatea

representativeclaim’sdeterminationtounadjudicatedclaimsanditsauthoritytoreplace

30Id.at78;seeSaks&Blanck,supranote7,at833–37.

31AggregatingforAccuracy,supranote2,at78–80.

57

anindividuallyadjudicatedoutcomewithanaggregatedone.32Adetaileddiscussionofthis

issueisbeyondthescopeofthisArticle.Assume,therefore,thatacourtmustchoose

betweensamplingaclaimforindividualadjudicationandpreservingtheclaimfor

assignmentofanaggregatedoutcomeextrapolatedfromthesampledclaims.33

Thisconstraintcanbemodeledusingatypeofexploration-exploitationtradeoffI

callreductivesampling,inwhichsamplingaunit—here,alegalclaim—meansreducing(or,

ashere,eliminating)itseligibilityforlaterextrapolationwithrespecttothatunit.34

InAggregatingforAccuracy,Ishowthat,usingthereductivesamplingframework

andstandardstatisticaltoolsforminimizingerror,inthecontextofaclassofhomogeneous

(orrelativelyhomogeneous)claims,accuracyismaximized,notbyadjudicatingeachofthe

claimsindividually(asisoftenviewedasideal),butratherbydeterminingindividual

outcomesforasmallrandomsampleofclaims,andassigningtheaverageofthesample

groupoutcomestotheremaining,non-sampled,claims.35Inparticular,Ishowthat,fora

classofNhomogeneousclaims,acourtmaximizesaccuracybysampling 𝑁claims,rather

thanallNclaims,forindividualadjudication.36Forexample,iftheclasscontains5,000

homogeneousclaims,acourtcanmaximizeaccuracybyrandomlysamplingabout70

32Id.

33Seeid.at80–81.

34Id.SeegenerallyHerbertRobbins,SomeAspectsoftheSequentialDesignofExperiments,58BULL.AM.MATHEMATICALSOC’Y527(1952);J.C.Gittins,BanditProcessesandDynamicAllocationIndices,41J.ROYALSTAT.SOC’Y:SERIESB(METHODOLOGICAL)148(1979).

35SeeAggregatingforAccuracy,supranote2,at81–82.

36Id.

58

claimsforindividualadjudication,assigningthe70individualoutcomestothesampled

claims,andassigningthearithmeticmeanofthe70sampleoutcomestotheremaining

4,930claims.

𝑁(orabout70intheexampleabove)isthenumberthatbalances,ontheone

hand,acourt’sinterestinobtaininginformationregardingthe“correct”outcomeofthe

homogeneousclaims(whichincreaseswithsamplesize),and,ontheotherhand,its

interestinpreservingclaimstowhichtoassigntheaccurateaggregatedoutcome.37The70

claimsinthesamplegroupreceiveindividualadjudicationsthataresubjecttosignificant

errorcausedbyjudgmentvariability,whereastheremaining4,930claimsinthe

extrapolationgroupareassignedaggregatedoutcomesthathavebeen“refined”by

repeatedadjudication—i.e.,outcomeswhosejudgmentvariabilityhasbeenreduced

significantlybyaveragingoverapproximately70repeatedadjudications.38

Now,toexaminesamplinginthecontextofheterogeneousclaims,consideran

additionalfactor:althoughthesamplingproceduredescribedabovereduceserrorcaused

byjudgmentvariability,assigningasingleaggregatedoutcome(e.g.,theaverageofthe

samplegroupadjudications)astheestimateofthe“correct”outcomesassociatedwitha

groupofheterogeneousclaims—claimsthatactuallyinvolvenumerousdistinct“correct”

outcomes—introduceserrorreflectingthedisparitiesbetweentheestimateandeachofthe

“correct”outcomes.39Thus,astheheterogeneityoftheclassincreases,thevalueof

37Seeid.at80–81.

38Id.at80–82.

39Id.at82–83.

59

assigningasingleaggregatedoutcomeastheestimateofthe“correct”outcomes

decreases.40Further,atsomedegreeofheterogeneity,thebenefitsofsampling,with

respecttojudgmentvariability,areoutweighedbythedetrimentsofsampling,withrespect

toclaimvariability.41

InAggregatingforAccuracy,Ishowthatifclaimvariabilityiszero,thentheoptimal

samplesizeis 𝑁,thehomogeneousoptimum;ifclaimvariabilityisgreaterthanjudgment

variability(e.g.,intermsofdamagesawarded),theoptimalsamplesizeisN,whichis

equivalenttoindividualadjudications;andfinally,ifjudgmentvariabilityisgreaterthan

claimvariability,thentheoptimalsamplesizeisbetween 𝑁andN,andcanbedetermined

byaparticularformula(derivedinAggregatingforAccuracy)involvingthenumberof

claimsintheclass,claimvariability,andjudgmentvariability.42

5. ImplicationsandConclusions

Theconclusionsexplainedabove,andderivedinAggregatingforAccuracy,have

importantimplicationsforacourt’streatmentofstatisticalsamplinginclassaction

litigation.Perhapsmostimportantly,acourtshouldnotassume,asmanycourtshave,that

samplingreducesreliability.Thediscussionabovemakesclearthatsamplingmayenhance

reliabilityaswellasefficiency.

Forexample,theseconclusionshaveimportantimplicationsinthecontextofclass

actionsbroughtunderRule23(b)(3)oftheFederalRulesofCivilProcedure.Inparticular,

40Id.

41Seeid.at82–83.SeegenerallySaks&Blanck,supranote7.

42SeeAggregatingforAccuracy,supranote2,at83.

60

classrepresentativeshaveproposedsamplingproceduresforpurposesoffulfillingthe

Rule’srequirementthat“thequestionsoflaworfactcommontoclassmembers

predominateoveranyquestionsaffectingonlyindividualmembers.”43Inaddressingthis

requirement,theclassrepresentativeshaveofferedsampling-basedmethodologiesatthe

classcertificationstageinattempttodemonstratethatliabilityanddamagescanbe

determinedonaclasswidebasis.44Courtsregularlyrejectsuchattempts,however,on

grounds—implicitlyorexplicitly—ofreliability.But,forthereasonsdiscussedabove,

courtsaregenerallynotjustifiedinassumingthatsamplingdiminishesreliability.

TheSupremeCourt,initsrecentdecisioninTysonFoods,Inc.v.Bouaphakeo,refused

toadopt“abroadruleagainsttheuseinclassactionsof...representativeevidence”—

specifically,evidencebasedon“arepresentativeorstatisticalsample”offeredtoestablish

classwideliability.45Instead,itheldthat“[w]hetherandwhenstatisticalevidencecanbe

usedtoestablishclasswideliabilitywilldependonthepurposeforwhichtheevidenceis

beingintroducedandon‘theelementsoftheunderlyingcauseofaction.’”46

TysonFoodsinvolvedclaimsbyTysonemployees,certifiedbythedistrictcourtasa

Rule23(b)(3)class,allegingthattheiremployer,Tyson,failedtopaycompensable

overtimewagesundertheFairLaborStandardsAct(FLSA)andIowalawfortimespent

43SeeComcastCorp.v.Behrend,133S.Ct.1426,1432(2013)(quotingFED.R.CIV.P.23(b)(3)).

44SeeComcastCorp.,133S.Ct.at1433–34;TysonFoods,Inc.v.Bouaphakeo,136S.Ct.1036,1043–44(2016).

45TysonFoods,Inc.,136S.Ct.at1046.

46Id.(quotingEricaP.JohnFund,Inc.v.HalliburtonCo.,563U.S.804,809(2011)).

61

“donninganddoffingprotectivegear.”47Thedistrictcourtsubmittedtheissuesofliability

anddamagestothejury,which“returnedaspecialverdictfindingthattimespentin

donninganddoffingprotectivegear...wascompensablework,”and“awardedtheclass

about$2.9millioninunpaidwages.”48Tysonappealed,arguing,interalia:“[T]heclass

shouldnothavebeencertifiedbecausetheprimarymethodofprovinginjuryassumedeach

employeespentthesametimedonninganddoffingprotectivegear,eventhough

differencesinthecompositionofthatgearmayhavemeantthat,infact,employeestook

differentamountsoftimetodonanddoff.”49TheEighthCircuitCourtofAppealsaffirmed

thejudgmentandtheaward.50

Inrulingthatthedistrictcourtdidnoterrincertifyingtheclass,theSupremeCourt

held:

In many cases, a representative sample is “the only practicable means tocollectandpresentrelevantdata”establishingadefendant’sliability.Manualof Complex Litigation §11.493, p. 102 (4th ed. 2004). In a case whererepresentative evidence is relevant inproving aplaintiff’s individual claim,that evidence cannot be deemed improper merely because the claim isbroughtonbehalfofaclass.TosoholdwouldignoretheRulesEnablingAct’spellucid instruction that use of the class device cannot “abridge . . . anysubstantiveright.”28U.S.C.§2072(b).51TheCourtcontinued:“Onewayforrespondentstoshow,then,thatthesample

relieduponhereisapermissiblemethodofprovingclasswideliabilityisbyshowingthat

47TysonFoods,Inc.,136S.Ct.at1042.

48Id.at1044.Notethataclassexpertrecommendedanawardofapproximately$6.7million.Id.at1052.

49Id.at1041.

50Id.at1044;Bouaphakeov.TysonFoods,Inc.,765F.3d791(8thCir.2014).

51TysonFoods,Inc.,136S.Ct.at1046.

62

eachclassmembercouldhavereliedonthatsampletoestablishliabilityifheorshehad

broughtanindividualaction.”52Itheldthat“[i]fthesamplecouldhavesustaineda

reasonablejuryfindingastohoursworkedineachemployee’sindividualaction,that

sampleisapermissiblemeansofestablishingtheemployees’hoursworkedinaclass

action.”53

TheCourtclarifieditsrulinginWal-MartStores,Inc.v.Dukes,54explainingthat“Wal-

Martdoesnotstandforthebroadpropositionthatarepresentativesampleisan

impermissiblemeansofestablishingclasswideliability.”55Wal-Martinvolvedaclassof

approximately1.5millioncurrentandformerfemaleemployeesalleginggender

discriminationunderTitleVII.56TheCourtexplainedthat“[t]heplaintiffsinWal-Martdid

notprovidesignificantproofofacommonpolicyofdiscriminationtowhicheachemployee

wassubject.”57Ultimately,theSupremeCourtrejectedplaintiffs’proposedmethodology—

proposed“asameansofovercomingth[e]absenceofacommonpolicy”58—bywhicha

sampleofclassmemberswouldbeselectedforindividualdeterminationsofliabilityand

backpay,andtheaggregateddamagesawardwouldbederivedbyextrapolatingthe

52Id.

53Id.at1046–47.TheCourtexplained:“Iftheemployeeshadproceededwith3,344individuallawsuits,eachemployeelikelywouldhavehadtointroduce[theclassexpert’s]studytoprovethehoursheorsheworked.Ratherthanabsolvingtheemployeesfromprovingindividualinjury,therepresentativeevidenceherewasapermissiblemeansofmakingthatveryshowing.”Id.at1047.

54564U.S.338(2011).

55TysonFoods,Inc.,136S.Ct.at1048.

56Id.

57Id.

58Id.

63

“numberof(presumptively)validclaims”fromthepercentageofthesampledclaims

determinedtobevalid,andthenmultiplyingthisnumberby“theaveragebackpayaward

inthesampleset.”59

TheCourtexplainedthatitsholdinginTysonFoodsis“inaccordwith”itsdecisionin

Wal-Mart:“SincetheCourtheldthattheemployeeswerenotsimilarlysituated,noneof

themcouldhaveprevailedinanindividualsuitbyrelyingondepositionsdetailingtheways

inwhichotheremployeeswerediscriminatedagainstbytheirparticularstore

managers.”60TheCourtexplainedthat“[b]yextension,iftheemployeeshadbrought1½

millionindividualsuits,therewouldbelittleornoroleforrepresentativeevidence,”and

that“[p]ermittingtheuseofthatsampleinaclassaction,therefore,wouldhaveviolated

theRulesEnablingActbygivingplaintiffsanddefendantsdifferentrightsinaclass

proceedingthantheycouldhaveassertedinanindividualaction.”61InTysonFoods,the

Courtheld:

[T]hestudyherecouldhavebeensufficienttosustainajuryfindingastohoursworkedifitwereintroducedineachemployee’sindividualaction.Whiletheexperiences of the employees inWal-Mart bore little relationship to oneanother, inthiscaseeachemployeeworkedinthesamefacility,didsimilarwork,andwaspaidunderthesamepolicy.62

59Id.(quotingWal-MartStores,Inc.v.Dukes,564U.S.338,367(2011)).

60TysonFoods,Inc.,136S.Ct.at1048.

61Id.

62Id.at1048.TheSupremeCourtheldthat,althoughthe“questionwhetheruninjuredclassmembersmayrecoverisoneofgreatimportance,”itisnot“aquestionyetfairlypresentedbythiscase,becausethedamagesawardhasnotyetbeendisbursed,nordoestherecordindicatehowitwillbedisbursed.”Id.at1050.TheCourtremandedthecaseforfurtherproceedingsconsistentwithitsopinion.Id.Significantly,ChiefJusticeRobertsexpressedconcern,inaconcurringopinionjoinedinpartbyJusticeAlito,that,sincethedistrictcourtmaybeunableto“fashionamethodforawardingdamagesonlytothoseclassmemberswhosufferedanactualinjury,”id.at1050,“itremainstobeseenwhetherthejuryverdictcanstand.”Id.at1053.Additionally,JusticeThomas,inadissentingopinionjoinedbyJusticeAlito,assertedthat“[t]heDistrictCourterredattheclasscertificationstagebyholdingthattheplaintiffssatisfiedRule23’spredominance

64

Thus,althoughtheSupremeCourtexplicitlyrefusedtoadopt“broadandcategorical

rulesgoverningtheuseofrepresentativeandstatisticalevidenceinclassactions,”the

TysonFoodsdecisionapprovestheuseofrepresentativeevidencetoestablishclasswide

liability.63

Distinguishtwotypesofsamplingmethodologiesthatputativeclassrepresentatives

haveattemptedtouseforestablishingclasswideliabilityanddamages:1)theuseof

“representativeevidence”offeredtothetrieroffactasprobativeofclasswideliability64or

damages,and2)theuseofrepresentativeadjudicationstoextrapolateoutcomesfornon-

sampled(i.e.,non-adjudicated)claims.65AlthoughtheSupremeCourtdidnotdistinguish

betweenthesetwotypesofsamplingmethodologies,itislikelythatTysonFoodsexpands

theabilityofclassrepresentativestouserepresentativeevidencetoestablishclasswide

liabilityinparticular.

Regardingtheuseofrepresentativeevidence,theCourtheld:“Arepresentativeor

statisticalsample,likeallevidence,isameanstoestablishordefendagainstliability.Its

permissibilityturnsnotontheformaproceedingtakes—beitaclassorindividualaction—

butonthedegreetowhichtheevidenceisreliableinprovingordisprovingtheelementsof

requirement.”Id.at1054.Accordingtothedissentingopinion,theissueof“whethereachemployeeworkedover40hourswithoutreceivingfullovertimepay”was“clearlyindividualized,”and,withrespecttothe“criticalissue”ofwhetherthe“individualizednatureofemployees’donninganddoffingtimesdefeatedpredominance,”id.,thedistrictcourterredbycertifyingtheclasswithoutgiving“properconsiderationtothesignificanceofvariabledonninganddoffingtimes.”Id.at1055.

63Id.at1049.

64Seeid.at1043.

65SeeAggregatingforAccuracy,supranote2,at70–72(citingcasesandliterature);see,e.g.,Ciminov.RaymarkIndus.,Inc.,751F.Supp.649,652–54(E.D.Tex.1990).

65

therelevantcauseofaction.”66TheCourt’sdecisioninTysonFoodsapprovestheuseof

statisticalsamplingtoestablishclasswideliability;buttheroleofreliabilityindetermining

whetheracourtpermitsstatisticalsamplingtoproveclasswideliabilityremainscentralto

theanalysis.

Theconclusionsexplainedabovesuggestthat,whileacourtshouldexaminethe

reliabilityofstatisticalsamplingfor,amongotherthings,methodologicalflawsandissues

relatedtothecohesivenessoftheclass,itshouldnotdiscountthereliabilityofstatistical

sampling—andrepresentativeevidenceinparticular—becauseofthesamplingitself.

Indeed,asexplainedabove,samplingmayimprovereliabilityaswellasefficiency.

Additionally,itisimportanttorealizethat,althoughthediscussionaboverelates

particularlytothereliabilitybenefitsofrepresentativeadjudications,theconclusions

generallyapplytorepresentativeevidenceaswell.Asexplained,repeatedadjudicationmay

improvereliabilitybyenablingacourt,inessence,toincorporateadditionalinformation

regardingaclassofclaims—e.g.,byaveragingovermultipleadjudicationsratherthan

relyingonasingleadjudication—andthusminimizeerrorcausedbyjudgmentvariability.

Repeatedadjudicationinaheterogeneousclasssimilarlyconfersreliabilitybenefits,aslong

astheerror-reducingbenefitsof“informationsharing”withrespecttojudgmentvariability

outweightheerror-inducingcostswithrespecttoclaimvariability.

Theuseofrepresentativeevidencesimilarlyimprovesreliability.Althoughthe

methodof“informationsharing”usingrepresentativeevidenceisdifferent—involving,for

66TysonFoods,Inc.,136S.Ct.at1046(citingFED.R.EVID.401,403,702)(emphasisadded).

66

example,theadditionalstepofprovidingtheinformationtothetrieroffact,ratherthan

incorporatingitintheoutcomedirectly(e.g.,byaveragingoverrepeatedadjudications)—

insofarasthetrieroffactincorporatestherepresentativeevidenceinitsdeterminationof

theoutcome,thereliabilitybenefitsofrepeatedadjudicationapplysimilarlytotheuseof

representativeevidence.67

TysonFoodsislikelytohaveasignificantimpactonclassactionlitigation.Putative

classrepresentativeswillbeencouragedtouserepresentativeevidencetoestablish

classwideliability,andperhapsdamagesaswell.Usingargumentsestablishing,for

example,that“eachclassmembercouldhavereliedonth[e]sampletoestablishliabilityif

67Inparticular,inAggregatingforAccuracy,anawardinaheterogeneousclassismodeledhierarchically:the“correct”awardsintheclassaredistributedaroundsomeglobalmean,whereaseachactualawardis“drawn”fromadistributionaroundeachclaim’s“correct”award.SeegenerallyAggregatingforAccuracy,supranote2,at82–83.Theformerdistributionrepresentsclaimvariability,whereasthelatterdistributionrepresentsjudgmentvariability.Inahomogeneousclass,replicationoffersaccuracybenefitsbyprovidingadditionalinformationregardingthe“correct”awardassociatedwiththereplicatedclaim,whichotherwisewouldbeobscuredbyjudgmentvariability.Inaheterogeneousclass,samplingoffersaccuracybenefits,withrespecttoacertainclaim,notbyprovidinginformationregardingthatclaim’s“correct”awarddirectly,butbyprovidinginformationregardingtheglobalmeanaroundwhichallofthe“correct”awardsaredistributed,andtherebyregardingthe“correct”awardforthesubjectclaimindirectly.Similarly,representativeevidence,suchasthetypeindisputeinTysonFoods(where,forexample,thesamplereflectsvariabilityofmeasureddonninganddoffingtimesratherthanjudgmentvariability),offersaccuracybenefits,withrespecttoacertainclaim,byprovidinginformationregardingtheglobalmeanaroundwhichthe“correct”awardsaredistributed,andtherebyregardingthe“correct”awardforthatclaiminparticular.Anotherwayofunderstandingthisisthrough“comparable-caseguidance”(CCG)methods,wherebyacourtusesinformationregardingawardsinpriorcomparablecasesasguidanceforafact-finder’sdeterminationofdamages.SeeHillelJ.Bavli,TheLogicofComparable-CaseGuidanceintheDeterminationofAwardsforPainandSufferingandPunitiveDamages,U.CIN.L.REV.(forthcoming2017)[hereinafterTheLogicofCCG].TheLogicofCCGexaminesthestatisticalmechanismbywhichCCGaffectsawards,andtheconditionsunderwhichsuchevidencewillimproveaccuracy.Inparticular,thepaperexplainsthat,undercertainmildbehavioralassumptions,theriskthatsuchevidencewouldreduceaccuracy—thaterrorresultingfromclaimvariabilityandbiaswouldoutweightheaccuracybenefitsofreducingjudgmentvariability—isminimal.Seeid.LikeCCG,representativeevidenceprovidesinformationregardingthedistributionof“correct”awardsforcomparableclaims,includingtheglobalmean,and,inturn,aboutthe“correct”awardforthesubjectclaim.

67

heorshehadbroughtanindividualaction,”putativeclassrepresentativeswillimprove

theirabilitytoestablishpredominanceunderRule23(b)(3).68

InlightofTysonFoods,acourtconsideringRule23(b)(3)certificationislikelyto

focusmoreheavilyandmoreexplicitlyonthereliabilityofrepresentativeevidenceand

statisticalsamplinggenerally.Althoughastatisticalsampleshouldbecarefullyscrutinized

todetect,amongotherthings,methodologicaldeficienciesandissuesregardingthe

cohesivenessoftheclass,samplingoftenimprovesthereliabilityoflegaloutcomes.

68TysonFoods,Inc.,136S.Ct.at1046.

68

III. TheLogicofComparable-CaseGuidanceintheDeterminationofAwards

forPainandSufferingandPunitiveDamages*

1. Introduction

Lawmakersandscholarshavestruggledtoaddresstheunpredictabilityofawards

forpainandsufferingandpunitivedamages.Jurorsareprovidedwithverylittleguidance

indeterminingsuchawards,andcourtslackobjectivestandardstoguidejurorsandreview

theirawards.Consequently,awardscanvarywildly.

Forexample,inawell-knowncaseinvolvingaclaimagainstBMWthattheauto

manufacturerhadfraudulentlysoldcarsasnewafterpaintingoverpartsthathadsuffered

corrosiondamageduetoacidrainexposurewhileintransittotheUnitedStates,an

Alabamajuryawardedthepurchasercompensatorydamagesof$4,000andpunitive

damagesintheamountof$4million.1However,inamateriallyidenticalcasebroughtby

anotherpurchaserinthesamecourtandbeforethesamejudge,butwithadifferentjury,

thejuryawardedasimilarlevelofcompensatorydamages,butnopunitivedamagesatall.

ThejuryfoundthatBMW’sbehaviordidnotrisetothelevelofreprehensibilitydeserving

ofpunitivedamages.2

*HillelJ.Bavli,TheLogicofComparable-CaseGuidanceintheDeterminationofAwardsforPainandSufferingandPunitiveDamages,U.Cin.L.Rev.(forthcoming,2017).TheauthorthanksDonaldRubin,DavidRosenberg,JosephBlitzstein,KennethShepsle,andYangChenfortheirhelpfulcomments,andJohnKennethFelterandtheUniversityofCincinnatiLawReviewEditorialBoardfortheirhelpfuledits.

1 The Alabama Supreme Court reduced the award to $2million, and the United States Supreme Courtultimatelyreversed,holdingthateven$2millioninpunitivedamageswasgrosslyexcessive.SeeBMWofNorthAmerica,Inc.v.Gore,517U.S.559(1996).

2SeeGeorgeL.Priest,IntroductiontoCassR.Sunsteinetal.,PUNITIVEDAMAGES:HOWJURIESDECIDE2-3(2002).

69

TheawardsintheBMWcase,likeothers—suchasamedicalmalpracticecase

resultinginapainandsufferingawardof$100million3oradeceptive-cigarette-marketing

caseresultinginapunitivedamagesawardof$28billion4—representsastarkdeviation

fromJusticeHolmes’scharacterizationofthelawasa“systematizedprediction.”5AsJudge

Niemeyercommentedregardingthe$100millionawardinEvans,“Becausethejurywas

givennorulenoranyrationalcriteriatoapplyinsettingtheamountofsuchanaward,but

toldsimplytodowhatitthoughtbest,thejuryrespondedwithaperceived‘measurement’

ofthepain,whichessentiallyamountedtoanemotionalresponse.”6Thejuryawardin

Evans—anawardtentimestheamountrequestedbytheplaintiff’sattorney,andonethat

thetrialjudgeultimatelyreducedto$3.5million—suggests,atleast,thatsuchawardsare

unpredictable.7RegardingtheawardinBullock,JudgeNiemeyercommentedthat“only

emotion,notaruleoflaw,couldjustifyimposinganawardof$28billion.Toconfirmthis,

3SeeEvansv.St.Mary’sHosp.ofBrooklyn,1A.D.3d314(N.Y.App.Div.2003).

4SeeBullockv.PhilipMorrisInc.,No.BC249171,2002WL31833905(Cal.App.Dep’tSuper.Ct.Dec.18,2002).

5OliverWendellHolmes,ThePathoftheLaw,10HARV.L.REV.457,458(1897).JusticeHolmesobservedthat“[p]eoplewanttoknowunderwhatcircumstancesandhowfartheywillruntheriskofcomingagainstwhatissomuchstrongerthanthemselves,andhenceitbecomesabusinesstofindoutwhenthisdangeristobefeared.Theobjectofourstudy,then,isprediction,thepredictionoftheincidenceofthepublicforcethrough the instrumentality of the courts.” Id. at 457. See also Paul v. Niemeyer, Awards for Pain andSuffering:TheIrrationalCenterpieceofourTortSystem,90VA.L.REV.1401,1402-04(2004).

6Niemeyer,supranote5,at1403.

7Id.at1403-04.

70

oneneedonlyconsiderthe‘mind’ofalegislativebodydevelopingaprospectiveruleoflaw

topunishsimilarconduct.”8

Indeed,numerousempiricalstudieshaveconfirmedsubstantialanecdotalevidence

thatawardsforpainandsufferingandpunitivedamagesarehighlyunpredictable.9

Commentatorshaveproposednumerousmethodstoaddressunpredictability,

includingvariousformsofawardschedulesthatbindorguideajuryinitsaward

determinationinlightofitsfindingsregardingcertainfacts,suchastheseverityofa

plaintiff’sinjuries.10Thus,“[s]cheduleswithcategoriesbasedoninjuryseveritytypically

providethemethodofclassification,andpriorawardsforinjurieswithineachcategory

providearangeofdamagesamounts.”11Thatis,“[a]lthoughreformsofthistypedifferin

8Id.at1409-10(notingthatthe“Californialegislaturehasfixedthemaximumfineforfalseadvertisingat$2500,”andthat“Congresshasfixedthemaximumfineforcorporations’violationsoffederaloffensesat$500,000ortwicethedefendant’sgainorthevictim’sloss”).

9SeeShariSeidmanDiamondetal.,JurorJudgmentsAboutLiabilityandDamages:SourcesofVariabilityandWaystoIncreaseConsistency,48DEPAULL.REV.301,317(1998)(concludingthatthereis“considerablevariationinbothjurorandjuryawards,”andthat“[a]substantialportionofthatvariationisnotpredictablefrommeasuresofeitherbackgroundorattitudinalindividualdifferencesacrossjurors”);RandallR.Bovbjergetal.,ValuingLifeandLimbinTort:Scheduling”PainandSuffering,”83NW.U.L.REV.908,919-24(1989)(concludingthat“[a]lthoughthemedian,andevenmean,awardsinagivencategorymaybeconsideredrelativelyreasonable,theseeminglyuncontrolledvariabilityofawardsiscauseforconcern—similartoanxietyaboutdrowninginapoolaveragingonlytwofeetindepth”);OscarG.Chase,HelpingJurorsDeterminePainandSufferingAwards,23HOFSTRAL.REV.763,768-69(1995)(citingstudies);DavidW.Leebron,FinalMoments:DamagesForPainandSufferingPriortoDeath,64N.Y.U.L.REV.256,259(1989)(concludingthat“tortawardsforeventhisrelativelysimplearea[ofdamagesforpainandsufferingpriortodeath]varysignificantlyandthatneitherthespecificfactsofthecasenordifferingtheoreticalviewsofthefunctionsoftheawardscanexplainsuchvariation”).SeealsoRichardAbel,GeneralDamagesAreIncoherent,Incalculable,Incommensurable,andInegalitarian(butOtherwiseAGreatIdea),55DEPAULL.REV.253,291-303(2006).ButseeYun-chienChangetal.,PainandSufferingDamagesinPersonalInjuryCases:AnEmpiricalStudy,UNIV.OFCHICAGOCOASE-SANDORINST.FORLAW&ECON.RESEARCHPAPERNO.749(April12,2016),http://papers.ssrn.com/abstract=2741180(concludingthat“painandsufferingdamagesinTaiwanaretoalargeextentstatisticallyandlegallypredictable”).

10SeeinfraSection3.

11MarkGeistfeld,PlacingAPriceonPainandSuffering:AMethodforHelpingJuriesDetermineTortDamagesforNonmonetaryInjuries,83CAL.L.REV.773,791(1995).

71

theirdetails,eachproceedsfromthepremisethatpriorpain-and-sufferingawardsfor

similarcasesprovidetheappropriatebasisforcomputingthepresentaward.”12Ultimately,

“[t]hejuryorreviewingcourtdetermineswheretheplaintiff’sinjuryfallsontheschedule,

andthescheduleprovidesarangeorspecifiedamountthatcanbebindingornonbinding

onjuriesorcourts.”13

Totheextentthatthesemethodspredeterminetheawardorrangeofawards,and

totheextentthattheybindthejuryratherthanguideit,theyhavebeencriticizedas

“eviscerat[ing]thevariouscontributionsthatjuriesmaketotheciviljusticesystem,”andas

being“inconsistentwiththebasictortprinciplethateachvictimisentitledtoanaward

tailoredtohisorhercircumstances,setbyalayjury.”14

Asimilarrecommendation,proposedpreviouslyinvariousforms,involves

“comparabilityanalysis.”Usingthisapproach,thecourt(perhapsbywayofanadversarial

processinvolvingthelitigationparties,andeventhetrieroffact)wouldfirstidentifya

universeofcomparablecases.Itwouldthenprovidethetrieroffactwithcertain

informationregardingtheawardsinthesecasesinthecontextofajuryinstructionoras

experttestimony,anditwouldinstructthetrieroffacttoarriveatadamages

determinationinlightoftheevidenceintroducedinthecase,andusingthecomparable-

caseinformation(or“prior-awardinformation”)asguidance.15Suchmethodsarebased,in

12Id.

13Id.

14DavidA.Logan,Juries,Judges,andthePoliticsofTortReform,83U.CIN.L.REV.903,942-43(2015).

15See infraSection3. Notethatthere issubstantialoverlapbetweensuchmethodsandthose involvingscheduling—particularlytotheextentthatschedulingmethodologiesinvolveprovidingafact-finderwithscenariosandassociatedawardvalues,basedoncomparablecases,asguidanceindeterminingdamages.

72

part,onempiricalstudiesconfirmingthattheyareeffectiveincontrollingoutlyingawards

andreducingawardvariabilitygenerally,evenusingpriorawardstoguide,ratherthan

bind,thetrieroffact.16

Forpurposesofclarity—becausetheterms“damageschedule”and“comparability

analysis”havebeenusedintheliteraturetomeanvariousthingsinvariouscontexts—Iwill

usetheterm“comparable-caseguidance”(CCG)torefertoschedulingorcomparability-

analysismethodsthatfulfillthreefundamentalrequirements:1)informationusedas

guidancemustbederivedfromprior“comparable”cases(asopposedto,e.g.,damage

schedulespredeterminedarbitrarilybyalegislativebody);2)comparable-case

informationmustbeconsideredbythetrieroffactinparticular(asopposedto,e.g.,a

reviewingcourt);and3)comparable-caseinformationmustbeusedasguidanceonly(as

opposedto,e.g.,imposingarangeoramountthatisbindingonthetrieroffact).17

WhileCCGmethodsallowforacase-by-caseanalysisthatremainsinthediscretion

ofthetrieroffact,theuseofpriorawardsitselfhasbeenattackedbasedonthree

fundamentalobjections.First,“[i]fthesystemhasbeenprovidingoverlyarbitrarypain-

and-sufferingawards,andifwehavenomethodfordeterminingtheappropriateawardin

thefirstinstance,whyshouldwemakepriorawardsthecornerstoneoffutureawards,”

since“[b]ydoingso,wemayensurethatlikecasesaretreatedalikeinthatallinvolve

SeealsoMichaelJ.Saksetal.,ReducingVariabilityinCivilJuryAwards,21LAWANDHUMANBEHAVIOR243,246(1997);Bovbjergetal.,supranote9,at953.

16See,e.g.,Saksetal.,supranote15,at249-55.

17Notethattheanalysishereinmayapplytoschedulingandcomparabilityanalysismorebroadly;however,IfocusonCCGmethodsinparticular.

73

inappropriatedamagesawards.”18Second,suchmethods“failtoaddressthefundamental

issueofhowoneshouldinitiallyassessthevalueofpain-and-sufferingdamages”orarrive

atanappropriatepunitivedamagesaward.19Third,thevalidityofsuchmethodsrelies

heavilyonthepresumptionthata“correct”setofcaseshasbeenidentified—thattheprior

casesidentifiedareindeedmaterially“comparable”tothecaseathand.20

Theseobjectionsboildowntoafundamentalconcern:ifourproblemisthe

unpredictabilityofawardscausedbyatrieroffact’sinabilitytoassessobjectivelythe

appropriatevalueofawardsforpainandsufferingorpunitivedamages,howisitbeneficial

toprovideatrieroffactwithinformationregardingdamagesawardedinpriorcasesthat

areseparateanddistinctfromthepresentcaseandthatpresumablysufferfromthesame

18Geistfeld,supranote11,at792(commentingthat“[t]hisrelianceuponpastawards...representsthemostproblematicaspectofthesereformproposals”).SeePeterH.Schuck,ScheduledDamagesandInsuranceContractsforFutureServices:ACommentonBlumstein,Bovbjerg,andSloan,8YALEJ.ONREG.213,218(1991)(“byusingearlierawardsasthefoundationfortheirnewsystemofdamagesscheduling,theyimpoundandthencompoundwhattheythemselvescharacterizeasthedistortionsofthepast,therebyprojectingthosedistortionsintothefuture”);seealsoRobertL.Rabin,ThePervasiveRoleofUncertaintyinTortLaw:RightsandRemedies,60DEPAULL.REV.431,448-49(2011)(“SchedulingproposalsofthiskindhavebeencriticizedbytortsscholarssuchasMarkGeistfeld,whopointedouttheseemingparadoxinrejectingunstructuredjurydecisionmakinginfavorofascheduledapproach,whichfromahorizontalequityperspectivetakesarbitrarypriorawardsasthecornerstoneforfutureawards,andfromaverticalequityperspectivetakestheorderingofmagnitudeinpastjuryawardsasanappropriatekeyforhierarchicalsortinginthedesignatedseverity-categoriesforfutureawards”).

19RonenAvraham,PuttingAPriceonPain-and-SufferingDamages:ACritiqueoftheCurrentApproachesandAPreliminaryProposalforChange,100NW.U.L.REV.87,104(2006)(discussingpain-and-sufferingawardsinparticular,andcitingW.KipViscusi,PainandSuffering:DamagesinSearchofASounderRationale,1MICH.L.&POL’YREV.141,168(1996)(“Thekeyissuetoberesolve[d]forpainandsufferingschedulesandscalesisthatestablishingsucharbitrarybenchmarksdoesnotresolvethemorefundamentalissueofhowoneshouldinitiallyassessthevalueofpainandsufferingdamages”)).

20SeeLogan,supranote14,at943-44(“While[anapproachallowingthefact-findertoconsiderarangeofpossibleawardsforguidance]wouldimprovepredictability,suchanapproachwouldonlybeasgoodasthequalityofthemethodologyforselectingwhichcaseswerefactuallysimilarenoughtobeincludedintherange”).

74

arbitrarinessthatwewishtoaddressinthepresentcase?Seemingly,thiswouldonly

compoundtheproblem.

Myaiminthisarticleistoaddresstheseobjectionsbyexplaininginsimplebut

formaltermshow,notwithstandingtheforegoingobjections,CCGmethodsreduce

unpredictabilityandimproveawardsforpainandsufferingandpunitivedamages

generallybyallowingforthesharingofrelevantinformationacrosscases.

Section2discussestheimportanceofreducingthevariabilityofawardsforpainand

sufferingandpunitivedamages.Section3providesbackgroundregardingcurrentmethods

andproposalsforreducingawardvariability.Section4explainshowCCGmethodsreduce

unpredictabilityandimproveawardsforpainandsufferingandpunitivedamages

generally.Section5discussesanumberofconsiderationsforidentifyingauniverseofprior

casesandfordistillinginformationfromsuchcasesforconsiderationbythetrieroffact.

Section6providesadditionallegalcontextforCCGmethodsandconcludes.

2. TheImportanceofReducingtheVariabilityofAwardsforPainand

SufferingandPunitiveDamages

TheSupremeCourthasrepeatedlyemphasizedtheimportanceofmaintainingfair

andconsistentawardsforpainandsufferingandpunitivedamages.Inarecentcase,for

example,theCourtstated:

Therealproblem, itseems, is thestarkunpredictabilityofpunitiveawards.Courtsof lawareconcernedwithfairnessasconsistency,andevidencethatthemedianratioofpunitivetocompensatoryawardsfallswithinareasonablezone,orthatpunitiveawardsareinfrequent,failstotelluswhetherthespread

75

between high and low individual awards is acceptable. The available datasuggestitisnot.21TheSupremeCourtcontinuedbydiscussingthe“inherentuncertaintyofthetrial

process”andtheresultinginconsistencyamongawardsincaseswithsimilarfacts.22In

examiningunpredictabilityasamatterofpolicy,ratherthanofconstitutional

significance,23theCourtemphasizedthattheunpredictabilityofhighpunitivedamage

awardsis“intensionwiththefunctionoftheawardsaspunitive.”24Itcommented:

Thus,apenaltyshouldbereasonablypredictableinitsseverity,sothatevenJusticeHolmes’s“badman”canlookaheadwithsomeabilitytoknowwhatthestakesareinchoosingonecourseofactionoranother.SeeThePathoftheLaw,10Harv.L.Rev.457,459(1897).Andwhenthebadman’scounterpartsturnup fromtimeto time, thepenaltyschemethey faceought to threaten themwithafairprobabilityofsufferinginlikedegreewhentheywreaklikedamage.Cf.Koonv.UnitedStates,518U.S.81,113(1996)(notingtheneed“toreduceunjustified disparities” in criminal sentencing “and so reach toward theevenhandedness and neutrality that are the distinguishing marks of anyprincipledsystemofjustice”).Thecommonsenseofjusticewouldsurelybarpenaltiesthatreasonablepeoplewouldthinkexcessivefortheharmcausedinthecircumstances.25

21ExxonShippingCo.v.Baker,554U.S.471,499(2008)(citingTheodoreEisenbergetal.,Juries,Judges,andPunitiveDamages:EmpiricalAnalysesUsingtheCivilJusticeSurveyofStateCourts1992,1996,and2001Data,3J.OFEMPIRICALLEGALSTUD.263,267(2006);ThomasH.Cohen,PunitiveDamageAwardsinLargeCounties,2001,U.S.DEPT.OFJUSTICE5(Mar.2005),http://www.bjs.gov/content/pub/pdf/pdalc01.pdf;BrianJ.Ostrometal.,AStepAboveAnecdote:AProfileoftheCivilJuryinthe1990s,79JUDICATURE233,240(1996);ErikK.Molleretal.,PunitiveDamagesinFinancialInjuryJuryVerdicts,28J.LEGALSTUD.283,333(1999)).

22Id.,554U.S.at500-01(quotingBMWofNorthAmerica,Inc.v.Gore,646So.2d619,626(1994)).

23TheSupremeCourthighlightedthat“theCourt’sresponsetooutlierpunitive-damagesawardshasthusfarbeenconfinedbyclaimsattheconstitutionallevel,andourcaseshaveannounceddueprocessstandardsthateveryawardmustpass.”Id.at501.InBaker,however,theCourtexaminedajuryawardforpunitivedamages“forconformitywithmaritimelaw,ratherthantheouterlimitallowedbydueprocess.”Id.at501-02.Inacting“inthepositionofacommonlawcourtoflastreview,”id.at507,itconsideredpunitivedamagesnotwithrespectto“theirintersectionwiththeConstitution,”butratherinrelationtothe“desirabilityofregulatingthemasacommonlawremedy.”Id.at502.

24Id.at502(emphasisadded).

25Id.at502-03.

76

TheSupremeCourtthusemphasizedthepointthatpredictability,consistency,and

fairnessarefundamentaltothedeterrenceobjectivesunderlyingpunitivedamages.This

sentimentandothersassociatedwiththeharmsofunpredictabilityhavebeenechoed

repeatedlybylowercourtsandscholars.

Forexample,inthecaseofPaynev.Jones,theSecondCircuitexplainedthepurposes

underlyingthedeviceofremittiturbyemphasizingtheharmfuleffectsofvariabilityand

outlyingawards,includingthoseassociatedwithover-deterrence:

Apart from impairing the fairness, predictability andproportionality of thelegalsystem,judgmentsawardingunreasonableamountsasdamagesimposeharmful, burdensome costs on society. As an initial matter, an excessiveverdictthatisallowedtostandestablishesaprecedentforexcessiveawardsinlatercases.Thepublicitythataccompanieshugepunitivedamagesawards,see,e.g.,HenryWeinstein,PhilipMorrisOrderedtoPay$28BilliontoSmoker,L.A.Times,Oct.5,2002,willencouragefuturejurorstoimposesimilarlylargeamounts. Unchecked awards levied against significant industries can causeseriousharmto thenationaleconomy.Productivecompaniescanbe forcedintobankruptcyoroutofbusiness.Municipalitiescanbedrainedofessentialpublicresources.Thethreatofexcessivedamages,furthermore,drivesupthecost of insurance premiums, deters both individuals and enterprises fromundertaking socially desirable activities and risks, and encouragesoverspendingon“sociallyexcessiveprecautions”that“cost[]morethanthereductionofharmproducedby[them].”A.MitchellPolinsky&StevenShavell,PunitiveDamages:AnEconomicAnalysis,111Harv.L.Rev.869,879(1998).Thepricesofgoodsandserviceswillrise,andinnovationwillbeinhibited.Seeid.at873.26

Courtshavesimilarlyemphasizedtheneedto“minimizethearbitraryvariancein

awardsboundtoresultfrom[the]throw-up-the-handsapproach”thatcourtsregularlyuse

indeterminingawardsforpainandsuffering.27Courtsandscholarshaverecognizedthe

26Paynev.Jones,711F.3d85,94(2dCir.2013).

27Jutzi-Johnsonv.UnitedStates,263F.3d753,759(7thCir.2001).

77

needtoaddressthis“standardless,unguidedexerciseofdiscretionbythetrieroffact,

reviewable...pursuanttonostandardtoguidethereviewingcourteither.”28

InGeressyv.DigitalEquip.Corp.,JudgeWeinsteinexplainedthecourt’sdecisionto

considerprior-awardinformation.Hediscussedconcernbythecourtsandlegislaturefor

the“virtuallyunbridleddiscretion”ofjuriesinawardingdamagesforpainandsuffering,

forwhichthereis“currentlynomeaningfulwaytomeasuresuchnon-quantifiablelosses

monetarily.”29AsProfessorOscarChaseexplained:

Variability is aproblemprimarilybecause itundermines the legal system’sclaimthatlikecaseswillbetreatedalike;thepromiseofequaljusticeunderlawisanimportantjustificationforourlegalsystem.Variabilityisalsoclaimedtocreateinstrumentaldefects;thatis,itmakesithardertosettlecases,thusadding unnecessary transaction costs to the tort system, and delayingpayment to needful plaintiffs. Unpredictability also leads to inefficienciesbecauseofover-orunder-precautionsbyaffectedindustriesandinsurers.30

Thus,whetherforpurposesoffairness,deterrence,oranotherobjective,courts

recognizetheimportanceofgeneratingconsistentandpredictabledamageawards.Indeed,

reducingvariabilityisfundamentaltoachievingreliablelegaloutcomes.InSection4,I

considervariabilityinthecontextoferrorgenerally,anddiscussinmoredetailwhatis

meantby“reliablelegaloutcomes.”First,however,Idescribecurrentmethodsand

proposalsforaddressingawardvariability.

28Id.

29Geressyv.DigitalEquip.Corp.,980F.Supp.640,656(E.D.N.Y.1997)(quotingLeslieA.Rubin,ConfrontingaNewObstacletoReproductiveChoice:EncouragingtheDevelopmentofRU-486ThroughReformofProductsLiabilityLaw,18N.Y.U.REV.L.&SOC.CHANGE131,146(1990-91)).

30Chase,supranote9,at769.

78

3. CurrentMethodsandProposalsforAddressingAwardVariability

AdditurandRemittitur

Courtsusetheproceduraldevicesofadditurandremittiturtoincreaseordecrease

anawardfoundtobeinsufficientorexcessive.Forexample,adefendantmayarguefora

newtrialbasedontheexcessivenessofthejury’saward.Ifthejudgeagrees,hemayoffer

theplaintifftoreducetheaward(remittitur)bysomeamountratherthanproceedingwith

anewtrial.31Appellatecourtsmayalsomodifyawardsandaddresschallengestoadditur

andremittitur.32

Althoughthedevicesofadditurandremittiturcan,intheory,addressthehighlevels

ofvariabilityassociatedwithawardsforpainandsufferingandpunitivedamages,in

practicetheycannot.First,additurandremittiturareprimarilyusedtoaddressexcessive

awards,andareinfrequentlyusedtoadjustinadequateawards.33Second,theseprocedures

aregenerallyreservedforonlythemostextremecases—theyareusedtoaddressextreme

outliersthataredeemedincorrect,ratherthanaddressvariabilityingeneral.34Third,

courtslackconsistentandprincipledproceduresforarrivingatbetteraward

determinationsthanjuries.AsJudgePosneropinedinJutzi-Johnsonv.UnitedStates,“[most

31 David Baldus et al., Improving Judicial Oversight of Jury Damages Assessments: A Proposal for theComparativeAdditur/RemittiturReviewofAwardsforNonpecuniaryHarmsandPunitiveDamages,80IOWAL.REV.1109,1118-20(1995).

32Id.Bothstateandfederalcourtsusetheadditurandremittiturprocedures;buttheSupremeCourthasdistinguishedadditur fromremittiturandheld that the formerprocedure isviolativeof thedefendant’sSeventhAmendmentrighttoatrialbyjury.SeeDimickv.Schiedt,293U.S.474(1934).

33SeeBaldusetal.,supranote31,at1119-20.

34Seeid.at1118-20.

79

courts]treatthedeterminationofhowmuchdamagesforpainandsufferingtoawardasa

standardless,unguidedexerciseofdiscretionbythetrieroffact,reviewableforabuseof

discretionpursuanttonostandardtoguidethereviewingcourteither.”35Further,courts

employthesedevicesinfrequently;36andwidespreadreplacementofjuryawardswith

judicialdeterminationswouldbeproblematicwithrespecttotheSeventhAmendmentand

fundamentalprinciplesoftortlaw.

DamageCaps.Similarly,damagecaps,whichplaceupperlimitsondamageawards

orcertaintypesofdamageawards,arewidelyrecognizedasaparticularlypoormethodfor

addressingvariability.37Theygiverisetoarangeofproblemsinherentincappingdamages

35Jutzi-Johnsonv.UnitedStates,263F.3d753,759(7thCir.2001).

36 See Baldus et al., supranote 31, at 1119-21 (citingMichael G. Shanley&Mark A. Peterson,PosttrialAdjustments to Jury Awards vii-viii, 43-47, RAND: THE INST. FOR CIVIL JUSTICE (1987)http://www.rand.org/content/dam/rand/pubs/reports/2007/R3511.pdf,andothers).Baldusetal.notethat“[o]nthebasisofpost-trialinformationinthe8801982-84CaliforniaandCookCounty,Illinoiscasesreportedin[theShanleyandPetersonstudy],weestimatethatincasesinvolvingadamagesaward,therewasaremittiturornewtrial6%ofthetimeandanaddituroranegotiatedincreaseintheawardin2-3%ofthecases.”SeealsoJosephSanders,WhyDoProposalsDesignedtoControlVariabilityinGeneralDamages(Generally)FallonDeafEars?(andWhyThisIsTooBad),55DEPAULL.REV.489,503(2006).

37Damagecapshaveservedastheprimarytooloftortreformaimingtoaddressvariability.Legislationforcapsboomedduringheightenedcallsfortortreforminthe1970sand1980s,andby1987,twenty-threestateshadinstitutedcapsrangingfrom$150,000to$1,000,000.Sanders,supranote36,at510.SeealsoStephenD.Sugarman,AComparativeLawLookatPainandSufferingAwards,55DEPAULL.REV.399,399-400(2006).Sincethen,numerousotherstateshaveadoptedcapsinvariouscontextsaswell.Sanders,supranote36,at510.Statutorycapsmaybeappliedtospecifictypesofdamages,suchaspunitivedamagesorgeneraldamages,ordamageawardsgenerally.SeeBaldusetal.,supranote31,at1121-23.Theymaybeapplied in particular contexts, such asmedicalmalpractice. See generallyKathryn Zeiler,Turning fromDamageCapstoInformationDisclosure:AnAlternativetoTortReform,5YALEJ.HEALTHPOL’Y,L.&ETHICS385(2005).Andtheyappearinavarietyofforms.Forexample,juriescanbeinformedofdamagecapsexplicitly,oracapcanbeappliedbythecourtonlyafterajuryhasexceededit.Ingeneral,juriesarenotinformedofcaps;but,regardless,jurorsmaybeawareofsuchmeasures.SeeSaksetal.,supranote15,at245.Undersomeproposals,ajudgemaybepermittedtoreduceawardsbelowthequantityprovidedbythelegislation;inothers,a judgemaybepermittedonly toreduceanaward to theestablishedquantity.SeeColleenP.Murphy,StatutoryCapsandJudicialReviewofDamages,39AKRONL.REV.1001,1002-08(2006).

80

(orcertaintypesofdamages)outright,independentofthecircumstancesofthecaseor

harmsufferedbytheplaintiff.

First,capsaddressonlythemostextremecases,andonlyexcessiveawards.Second,

capsarelikelytoplayaroleprimarilyincasesinwhichtheharmisverysevere.If,for

example,thereisacapof$1million,thecapmaynotaffectanindividualwhosuffereda

brokennose,butwouldgreatlyaffectanindividualwhosufferedparalysis.Thecapis

unlikelytoaddressvariabilityintheformercase,whilepossiblycausinganinappropriately

lowawardinthelatter.38Thus,asidefrombiasingtheaward,capsdistortincentivesto

litigatetortclaimsbasedonsevereharm.Further,capscancausesuboptimalrisk-takingby

dilutingthedeterrenceeffectofpunitivedamagesandtortlawgenerally.39Third,anumber

ofstudieshaveconcludedthatcapsmayinfactexacerbatethevariabilityproblemrather

thanaddressit.Inparticular,jurorswhoareawareofacapmayanchortoit,andtheir

awardsmaygravitatetoit.“Thus,thosewiththemostsevereinjuriesandlosseswillbe

unfairlydeprivedofcompensationbycaps.Inaddition,byservingasapsychological

anchor,capsappearlikelytofurtherexaggeratetheerrorofovercompensatingthosewith

smallerlosses.”40

38SeeBaldusetal.,supranote31,at1121-23.

39SeeSanders,supranote36,at509-11.Constitutionalityissuesmayariseaswell.Opponentsofcapshaveargued,forexample,thatcapsareviolativeofaplaintiff’srighttotrialbyjury.SeeZeiler,supranote37,at387.

40Saksetal.,supranote15,at245.SeealsoVerlinB.Hinsz&KristinE.Indahl,AssimilationtoAnchorsforDamageAwardsinaMockCivilTrial,25J.OFAPPLIEDSOC.PSYCHOLOGY991(1995).

81

But,notwithstandingtheforegoingproblems,damagecapsareused,due,inpart,to

theiradministrativeconvenienceandtheirabilitytocontrolinsurancerates.41

Comparable-CaseInformation.Finally,anumberofcourtsandcommentators

haveproposedmethodsinvolvingtheuseofinformationregardingawardsinprior

“comparable”cases.Suchproposalshaveappearedinvariousforms.Onesetofapproaches

buildsoncourts’regularuseofcomparabilityreview,wherebytrialandappellatecourts

considerawardsincomparablecasesintheirreviewofatrieroffact’sawardfor

excessiveness.42Theseproposalsattempttodevelopamorestructuredframeworkforthe

court’scomparativeanalysis—forexample,institutingamethodbywhichacourtidentifies

auniverseofcomparableclaims;examineswhetherthecurrentaward“deviatesmaterially

fromwhatwouldbereasonablecompensation”(touselanguagefromaNewYorkstatute

uponwhichsuchproposalsseemtobuild);43and,ifso,determinesasuitableadjustmentof

thejuryaward.44

Thesereview-basedmethodshavetheadvantageofinvolvingonlymodest

modificationstocurrentwell-acceptedmethods,buttheyarenotideal.Specifically,they

incorporatecomparable-caseinformationonreviewratherthanintheawarddirectly.As

such,theyonlyfocusonoutlyingawardsratherthanonvariabilityingeneral.Moreover,

41SeeSanders,supranote36,at511.

42SeeBaldusetal.,supranote31,at1134.

43N.Y.C.P.L.R.5501(c)(McKinney1995).

44SeeBaldusetal.,supranote31,at1134.SeealsoSanders,supranote36,at503.

82

regularuseofsuchmethodsarguablyleadstoconstitutionalityandtortissuesassociated

withthereplacementofajury’sdiscretionwiththatofthejudge.45

AsindicatedintheIntroduction,anothersetofproposalsusesvarioustypesof

schedulestobindorguidethetrieroffactinitsdeterminationofcertaintypesofdamage

awards.46Theseproposals,totheextenttheyinvolvebindingatrieroffactto

predeterminedvaluesorrangesofvalues,havebeencriticizedasimpingingonthetrierof

fact’sdiscretion,andas“inconsistentwiththebasictortprinciplethateachvictimis

entitledtoanawardtailoredtohisorhercircumstances,setbyalayjury.”47Totheextent

thatsuchmethodsbindratherthanguidethetrieroffact,andtotheextentthatthey

involveawardvaluesthatarepredetermined,forexample,byalegislativebodyinadvance

ofthecase,suchmethodsarearguablyproblematic.

Finally,anumberofproposals—overlappingwiththecategoryaboveregarding

schedules—involvecomparabilityanalysis,wherebythecourt(perhapsbywayofan

adversarialprocessinvolvingthelitigationparties,andeventhetrieroffact)identifiesa

setofmaterially“comparable”cases,providesthetrieroffactwithcertaininformation

regardingtheawardsinthesecasesinthecontextofajuryinstructionorpresentedas

45SeeColleenP.Murphy,JudicialAssessmentofLegalRemedies,94NW.U.L.REV.153,192(1999);J.PatrickElsevier,Out-of-Line:FederalCourtsUsingComparabilitytoReviewDamageAwards,33GA.L.REV.243,258-59 (1998) (“Comparability analysis [on review] requires a court to engage in several subjective,factintensiveinquiries...Yet,byassertingthatcomparabilityprovidesanobjectiveframeworktoreviewcompensatorydamageawards,courtsareabletosubstitutetheirownassessmentofwhattheappropriatedamageawardshouldbe,whileskirtingtheSeventhAmendment’sproscriptionagainstreexaminationofissuesoffact”).

46See,e.g.,Bovbjergetal.,supranote9,at945.

47Logan,supranote14,at943.

83

experttestimony,andinstructsthetrieroffacttodetermineanappropriatedamages

awardinlightoftheevidenceintroducedinthecase,andusingthecomparable-case

informationasguidance.48

AssuggestedintheIntroduction,suchmethods—andCCGmethodsinparticular—

avoidmanyoftheproblemscausedbyothermethods,suchasdamagecapsandbindingor

predeterminedschedules.Thesemethodsallowforaflexiblecase-by-caseanalysisinthe

discretionofthetrieroffact,whilestillprovidingsubstantialguidance.Nevertheless,such

methodshavebeenattackedonthegroundsthat,insteadofaddressingthefundamental

issueofassessingthevalueofpain-and-sufferingdamagesoranappropriatepunitive

damagesaward,theymaycompoundtheunpredictabilityofdamageawardsbyproviding

thetrieroffactwithinformationregardingdamagesawardedinseparateanddistinctprior

casesthatpresumablysufferfromthesamearbitrarinessthatthecourtaimstoaddressin

thepresentcase.49Inthefollowingsections,Iaddresstheseobjections.

4. TheLogicofCCG

Inthissection,Iexplainwhy,notwithstandingtheobjectionsdelineatedabove,CCG

methodsreduceunpredictabilityandimprovethereliabilityofawardsforpainand

sufferingandpunitivedamagesgenerally.Ibeginbydiscussingwhatismeantby

48See,e.g.,Chase,supranote9,at777-90(proposingthatcourtsinformjurorsof“therangeofawardsmadebyotherjuriesinthesamestateforsuchdamagesduringacontemporaneoustimeperiod,”tobeprovidedas non-binding guidance in the court’s charge to the jury in the form of a “chart constructed to allowcomparison with roughly similar cases in which plaintiffs’ verdicts were recovered.” Id. at 777.). SeegenerallyBaldusetal.,supranote31,at1123-24;Saksetal.,supranote15,at246;Sanders,supranote36,at 506-7; JACOB A. STEIN, STEIN ON PERSONAL INJURY DAMAGES TREATISE §8:32 (3d ed. 2015) (discussingAmericanBarAssoc.,ReportoftheActionCommissiontoImprovetheTortLiabilitySystem10-15(1987));Jutzi-Johnsonv.UnitedStates,263F.3d753,758-61(7thCir.2001).

49SeeGeistfeld,supranote11,at792;Avraham,supranote19,at104;Logan,supranote14,at943-44.

84

“reliability.”IthenintroducetheroleofCCGinreducingvariabilityanderrorin,andthus

improvingthereliabilityof,legaloutcomes.

4.1 BiasandVariance:TheElementsofError

Section2makesclearthatcontrollingthevariabilityofawardsforpainand

sufferingandpunitivedamagesisimportant.Butreducingvariabilitydoesnotnecessarily

meanimprovingtheaward.Forexample,itislikelyunwisetoencouragejurorstoanchor

toanarbitraryvalue,notwithstandingassociatedvariabilitybenefits.Infact,variability

couldbezeroifthecourtweretodictateanawardwithoutregardtotheparticularsofthe

case.

Courtsandscholarsarecorrecttobeconcernedaboutthehighvariability

associatedwithawardsforpainandsufferingandpunitivedamages.Butgoodpolicy

requiresstepstowardreducingsuchvariabilityonlyinsofarastheyimproveawards

generally.

Thus,forpurposesofconsideringtheconceptsof“error”and“accuracy”inthe

contextofdamageawards,assumethatthereissome“correct”awardthatcould,intheory,

bedeterminedasafunctionofperfectinformationregardingthestateoftheworld,

includingthefactsofthecaseandapplicablelawandnorms.50Ofcourse,weneitherhave

perfectinformationnorknowthecorrectaward.Instead,thecourtasksajurytoarriveat

anaward,whichwillserveasanestimateofthecorrectoutcome.

50SeeHillelJ.Bavli,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyinClassActionLitigation,14LAW,PROBABILITY&RISK67,74-78(2015). Wecansimilarlyconsiderarangeordistributionof “correct”awardsthatreflects,forexample,uncertaintyregardingthelaw.Id.at74n.24.

85

Moreconcretely,letusconsiderthecorrectawardinagivencase—say,forexample,

theBMWcasedescribedintheIntroduction—tobethemeanofthepopulationofpossible

awardsthatwouldemergefromadjudicatingthecaserepeatedlyundervariousconditions

(e.g.,beforedifferentjudgesandjuries,bydifferentattorneys,withdifferentpermutations

offacts,etc.).51Asingletrialthusgeneratesasamplefromthepopulationandanestimate

ofthecorrectaward.52Calltheactualawardan“estimate,”andtheprocedurethat

generatestheestimatean“estimator.”53Wecanthendefine“error”intermsofdistance,

and(equivalently)“accuracy”intermsofproximity,betweentheestimateandthecorrect

award.54

Idefinethereliabilityofalegalprocedureastheaccuracyoftheoutcome(here,the

award)thatwecanexpectbyfollowingtheprocedure.Ifweexpectthatacertainlegal

51Id.(citingMichaelJ.Saks&PeterDavidBlanck,JusticeImproved:TheUnrecognizedBenefitsofAggregationand Sampling in the Trial of Mass Torts, 44 STAN. L. REV. 815, 833-34 (1992)). For simplicity, I ignore thepotential forbias. AsnotedinBavli, supranote47,“[i]tmaybemoreintuitivetoconsidertheconceptofa‘correct’verdict inthecontextofacriminaltrial.Consider, forexample,theO.J.Simpsonmurdertrial.Pollsshowthatover50%ofAmericansbelievethatthejuryarrivedatanincorrectverdict.Implicitinthepublic’sdisagreementwiththeverdictisanassumptionthatthereexistsa‘correct’outcome.Theframeworkdescribedaboveisintuitive:hadthejuryknownthetruefactsofthecase,andhadtherebeennoambiguityregardingtheapplicationofthelawtothefactsofthecase,thejurywouldhavearrivedatthecorrectconclusion.Butgivenambiguityregardingeitherthefactsofacaseorthestateofthelaw,itisunclearwhetheracriminaldefendantinfactsatisfiedtheelementsofthecrimecharged;andthejurymustarriveataverdict—‘guilty’or‘notguilty’—which serves as an estimate of the correct outcome, ‘guilty’ or ‘not guilty.’” Id. at 74 n.29. Similarly, thedeterminationof a civil damagesaward (or,e.g., aprison sentence) canbeunderstoodas anestimateof acorrectoutcome.Arguablycourtsimplicitlyacknowledgethischaracterizationwhen,forexample,acourtfindsajuryawardtobeexcessiveorinadequate.Finally,theformulationofthecorrectawardasthemeanofthepopulationofpossibleawardsoverrepeatedadjudicationsisintendedtocapture,e.g.,thestateofthelawasunderstoodbyvariousjudgesacrosstherelevantsociety,andthenormsofthetimeandfactsofthecaseasunderstoodbyvariouscombinationsofjurorsintherelevantsociety.

52Seeid.at74-75;Saks&Blanck,supranote51,at833-34.

53Id.Theestimatorcanbeunderstoodasthelegalprocedure—includingtherulesgoverningthetrial,theroleofthejudge,thejurydeliberation,etc.—thatgeneratesthelegaloutcome,which,inourexample,isthejury’spunitivedamagesaward.

54Seeid.at74-78.

86

procedurewillproduceanaccurateaward,wesaythattheprocedure(or,forsimplicity,

theaward)is“reliable.”55

Instatisticstherearetwosourcesoferror:biasandvariance.Ifanestimatoris

“unbiased,”thenitwillgeneratethecorrectvalueonaverage.Ifitis“biased,”thenitwill

generatetheincorrectvalueonaverage,andthe“bias”reflectsthedistancebetweenthe

valuetheestimatorgeneratesonaverageandthecorrectvalue.Biasisthereforeasource

oferror.Notethatitispossiblethatanunbiasedestimatorwillnevergeneratethecorrect

value—tobeunbiasedisonlytogeneratethecorrectvalueinexpectation,oronaverage.

Additionally,althoughunbiasednessisgenerallyunderstoodasagoodcharacteristicforan

estimatortohave,itdoesnotindicatelackoferror,sincethevaluesgeneratedbythe

estimatorcanvarywildlyaroundthecorrectvalue.Forexample,ifthecorrectpunitive

damagesvalueintheBMWcaseaboveis$100,000,thenrepetitionsofan“unbiased”

adjudicationmaygenerateestimatevalues(i.e.,damageawards)of$0,$50,000,$150,000,

and$200,000,whichareindeedcenteredatthecorrectvalue,$100,000,buttheawardsare

highlydispersedaround$100,000.Wewould,forexample,preferthatrepeated

adjudicationsgeneratethevalues$90,000,$95,000,$105,000,and$110,000;oreven

better,$100,000,$100,000,$100,000,and$100,000.

Thus,thesecondsourceoferroris“variance,”whichisameasureofdispersion.In

particular,ifanestimatorentailsahighlevelofvariance,thenitwillgenerateestimates

thatarehighlydispersedarounditsmean,oraverage,value.Iftheestimatorentailsahigh

55Seegenerallyid.

87

levelofvariance,butisunbiased,thenitwillgenerateestimatesthatarehighlydispersed

aroundthecorrectvalue.Inthiscase,wesaythattheestimatoris“unbiased,”butthatit

lacks“precision.”Iftheestimatoris“precise”but“biased,”thenitgeneratesvaluesthatare

tightlycenteredaroundthewrongvalue—anundesirablecircumstance.Ifanestimatoris

“precise”and“unbiased,”thenitwillgenerateestimatesthatarecloseinproximitytothe

correctvalue,andwesaythatitis“accurate.”

Thus,intheBMWexampleabove,theawards$90,000,$95,000,$105,000,and

$110,000reflectgreaterprecisionthan$0,$50,000,$150,000,and$200,000.And,the

awards$100,000,$100,000,$100,000,and$100,000reflectevengreaterprecision.

Moreformally,letαbethe“correct”awardand𝛼arandomvariabledefinedbythe

actualaward.Let𝐸(𝛼)betheexpectationof𝛼.Inotherwords,𝛼willequal𝐸(𝛼)on

average;andifwerepeatthetrialmanytimesthentheaverageofthepunitivedamage

awardswillbeapproximately𝐸(𝛼).

Biasisdefinedasthedifferencebetweentheexpectationoftheestimator𝛼andthe

correctawardα(theobjectwearetryingtoestimate).Moreformally𝐵𝑖𝑎𝑠 = 𝐸 𝛼 − 𝛼,and

itissaidthattheestimator𝛼is“unbiased”if𝐸 𝛼 = 𝛼.

Additionally,let𝑉(𝛼)bethevarianceof𝛼,wherevarianceisdefinedinthestandard

way.Thatis,

𝑉 𝛼 = 𝐸 𝛼 − 𝐸 𝛼 ) =1𝑛 𝛼" − 𝐸 𝛼 )

+

"23

.

88

Thevarianceoftherealizedawardsisthustheaverageofthesquaredifferences

betweentheawardsandtheaverageoftheawards.Forexample,thevarianceof$0,

$50,000,$150,000,and$200,000is3^( $0 − $100,000 ) + $50,000 − $100,000 ) +

$150,000 − $100,000 ) + $200,000 − $100,000 )).Thestandarddeviationof𝛼isthe

squarerootofthevariance:𝑆𝐷 𝛼 = 𝑉(𝛼).

Nowletusformallydefineerrorintermsofbiasandvariance.Therearemanyways

tomeasureerror(justastherearemanywaystomeasure,forexample,dispersion).For

instance,wemightdefineerrorbytherawdifferencesbetweentheestimatesandthe

correctvalue.Butjustasitisinconvenientforscientiststodefinedispersionbycalculating

therawdifferencesbetweentheestimatesandthemeanoftheestimates,56similar

inconveniencesarisefromdefiningerrorintermsofrawdifferences.Itturnsoutthata

convenientandoftensuitablemeasureoferroristhewell-acceptedmeansquarederror

(MSE),definedas𝑀𝑆𝐸 𝛼 = 𝐸[ 𝛼 − 𝛼 )].Inwords,wefindtheexpectationofthesquare

differencebetweentheestimatorandthecorrectvalue.

Now,usingMSEasourmeasureoferror,itiseasytoshowthat:

𝑀𝑆𝐸 𝛼 = 𝐸[ 𝛼 − 𝛼 )] = 𝑉𝑎𝑟𝑖𝑎𝑛𝑐𝑒 + 𝐵𝑖𝑎𝑠)

Thatis,error(definedhereasMSE)canbeseparatedintotwoelements:biasand

variance.57

56Forexample,usingsuchdifferencesresultsinthe“canceling”ofpositiveandnegativevalues.

57SeeSHARONL.LOHR,SAMPLING:DESIGNANDANALYSIS,§2.2(2ded.2010),foramathematicalproof,andalengthierdiscussion,ofthebias-variancedecomposition.

89

4.2 ReducingVariabilityandErrorwithPrior-AwardInformation

Letusviewacivilcaseasanestimationproblemwithintheframeworkdescribed

above.Thatis,foreveryclaimletusassumethatthereisa“correct”damagesaward(or,in

particular,a“correct”awardforpainandsufferingorpunitivedamages)α,definedasthe

meanofthepopulationofawardsthatwouldresultfromrepeatedadjudications,as

describedabove(andthatwouldreflect,forexample,perfectinformationregardingthe

applicablelawandthefactsofthecase).58Asdiscussed,wecannotknowthecorrectaward

andmustthereforearriveatasuitableestimate𝛼oftheunknowncorrectaward.

Thus,letusassume(forsimplicityratherthannecessity)that,onaverage,the

estimatewillequalthecorrectaward,butthatitentailssomedegreeofvariabilityσ2

aroundthecorrectaward.Thatis,𝛼isdistributedwithmeanαandvarianceσ2.

Notationally,𝛼~(𝛼, 𝜎)).

Ourgoalistoarriveatadamagesestimatethatisclosetothecorrectaward—thatis

“accurate”inthesenseofminimizingerror,asdefinedaboveby𝑀𝑆𝐸 𝛼 = 𝐸[ 𝛼 − 𝛼 )].

Instatistics,andthesciencesgenerally,determiningagoodestimatorforan

unknownquantityisveryimportantandisthesubjectofmuchresearch.Letususean

analogytobetterunderstandtheproblemandpossiblesolutions.Considerastudyby

statisticiansBradleyEfronandCarlMorris,inwhichtheyrecordedbattingaveragedatafor

eachofeighteenmajorleaguebaseballplayersthroughhisfirstforty-fiveofficialatbatsof

58SeeBavli,supranote50,at74-78.Wecandefinethe“correct”awardusingothermeasuresofcentraltendency,suchasthemedian,dependingonhowwewanttocharacterizeit.Themean,forexample,ismoresensitivethanthemediantoextremeawards.Forsimplicity,ouranalysisinthisarticlefocusesonlyonthemean.

90

the1970season.59Supposeourgoalis(astheirgoalwas)toestimatethebattingaverages

thatwouldemergeforeachplayerduringtheremainderoftheseason,usingonlythedata

collected.Letusfocusononeplayer,RobertoClemente,whosebattingaverageforhisfirst

45atbatswas.400.60

Initially,itmightseemelementarythataplayer’sbattingaverageforthefirstforty-

fiveatbatswouldserveasthebestestimatorforhisbattingaveragefortheremainderof

theseason.Thus,ourestimateofClemente’sremainder-of-the-season(or“after-forty-five”)

battingaveragewouldbe.400.Butitcanbeshownmathematicallyandempiricallythat

thisisnotinfactthebestestimator.Inparticular,thisestimator—Clemente’sbatting

averageforhisfirstforty-fiveatbats—canbeimprovedbyincorporatinginformation

regardingthebattingaveragesforthefirstforty-fiveatbatsoftheotherseventeen

players.61Butwhyshouldthisbe?WhyshouldthebattingaveragesofMunsonor

Kessinger,whohad“first-forty-five”battingaveragesof.156and.289,respectively,have

anyinfluenceonourestimateofClemente’safter-forty-fivebattingaverage?Usingother

players’first-forty-fivebattingaverageswouldseemobviousintheabsenceofClemente’s

first-forty-fiveaverage;butwehaveClemente’sfirst-forty-fivebattingaverage—andwhat

couldbebetterthanthat?

Theexplanationliesintheinformationweobtainfromthefirst-forty-fivebatting

averagesoftheotherseventeenplayers.First,supposeweknownothingaboutbatting

59SeeBradleyEfron&CarlMorris,DataAnalysisUsingStein’sEstimatorandItsGeneralizations,70J.OFTHEAM.STATIST.ASS’N311,312-14(1975).

60Id.at313.

61Seeid.at312-14.

91

averages.Itseemsintuitivethat,inestimatingClemente’safter-forty-fivebattingaverage,it

ishelpfultoknowwhetherhisfirst-forty-fivebattingaverage,.400,is“low,”“high,”or

“average.”Inparticular,itseemshelpfultoknowthat.400isanextremelyhighbatting

average.Asanextremeexample,ifwearetoldthat,inagivenperiod,aparticularbatting

averageisthehighesteverachievedbythatplayer,orinbaseballingeneral,wouldwestill

usethisaverageasourbestestimateoftheplayer’saverageinthenextperiod?Wewould

bewisetoincorporatethisinformation,sincetheplayerisunlikelytobreakbaseball’sall-

timerecordtwiceinarow.Morespecifically,theusefulnessofother-playerinformation

canbeexplained,inpart,byconsideringtheoldidea(duetoSirFrancisGalton)of

“regressiontowardthemean,”which,initssimplestform,statesthatifweobtaina

relativelyextrememeasurement,thenuponre-measurement,wearelikelytoobtainanew

measurementclosertoaverage.62Thereasonissimple:anextrememeasurementcanbe

attributedtoacombinationoftwothings—thecharacteristicsofthethingmeasured(e.g.,

theskillofabaseballplayerwhowasinfactafarsuperiorbatterthanhispeers)and

randomvariation,or“luck.”Totheextentthattheextrememeasurementisdueto

randomness,itislikelytobelessextremeuponre-measurement.Similarly,information

regardingotherplayers’performanceprovidescontextforClemente’sperformance,andan

estimatorthatincorporatessuchinformationislikelybetterthanonebasedsolelyonthe

player’sindividualperformance.63

62SeegenerallyStephenM.Stigler,Darwin,GaltonandtheStatisticalEnlightenment,173J.R.STATIST.SOC’YA469(2010).

63Wecanconsidertwosourcesofvariability:thereisvariationamongtheplayers(somearebetterthanothers,forexample)and,givenaparticularplayer(andgivenhismeanbattingaverageinparticular),thereisvariationintheplayer’sbattingaverage.“Regressiontowardthemean”thusoperatesintwoways:First,

92

Inshort,ifwebelievethattheplayers’battingaveragesaresomehowbound

together,wecanmakesignificantestimationimprovementsbyincorporatingother-player

information.64

Returningtothelegalcontext,thevalueofadamagesaward(or,specifically,an

awardforpainandsufferingorpunitivedamages)canbeunderstoodasattributableto1)

theapplicationofthelawtotheparticularfactsofthecase,and2)randomvariation.In

ordertogainadeeperunderstandingofthebenefitsofCCG,letusmodelanaward

“hierarchically”toaccountforbothoftheforegoingelements.Asabove,weassumethatthe

actualaward𝛼isdistributedwithmeanαandvarianceσ2.Butnow,letusassumethatαis

itselfrandomratherthanconstant,andthatitisdistributedwithmeanµandvarianceτ2.

Notationally,𝛼~(𝛼, 𝜎))and𝛼~(𝜇, 𝜏)).Inwords,thehierarchicalmodelincorporates

variabilityontwolevels.First,theupperlevel 𝛼~ 𝜇, 𝜏) indicatesthatcasesare

heterogeneous—thattherecanbefactualvariabilityacrosscases,andthatthepresentcase

maybemateriallydifferentfrompriorcases.Irefertothisformofvariability,τ2,as“claim

variability.”65Second,thelowerlevel(𝛼~ 𝛼, 𝜎) )indicatesthat,giventhefactsofany

onthelattersource,pullingaplayer’sbattingaverageintowardhismeanbattingaverage;andsecond,onthe former source, pulling a player’s batting average in toward the league’s mean batting average.Incorporatinginformationregardingthedistributionofplayersallowsustoaccountforthepulltowardtheleague’smeanbattingaverage.

64Efron&Morrisdiscussconditionsunderwhichestimationimprovementscanbemadeevenwhentheestimands—thethingswewishtoestimate—arenotboundtogether.SeegenerallyEfron&Morris,supranote59;BradleyEfron&CarlMorris,Stein’sParadoxinStatistics,236SCIENTIFICAMERICAN119(1977).

65SeeBavli,supranote50,at74-78.

93

singlecase(forexample,thepresentcase),thereisvariability(i.e.,randomness)inthe

determinationofdamages.Irefertothisformofvariability,σ2,as“judgmentvariability.”66

Again,ourgoalistoarriveatagoodestimate𝛼ofα.Ononehand,wewantthe

estimatetoreflectthefirstformofvariability—thatmeaningfulfactualdifferences(e.g.,

betweenthepresentcaseandpriorcases)shouldresultindifferentawards.Ontheother

hand,wewanttominimizethesecondformofvariability—therandomnessassociated

withthedeterminationofdamagesinagivencase.Sohowdoesincorporatingprior-award

informationhelp?Unfortunately,sincewegenerallydonotknowthecorrectdamages

award,itisimpossibletodistinguishtheformerformofvariabilityfromthelatter.Inother

words,atradeoffarisesbetweenminimizingbias—thatourestimateddamagesaward

should,onaverage,beasclosetothecorrectdamagesawardaspossible,reflectingthe

formerformofvariability—andminimizingvariance,thedegreeofrandomnessassociated

withtheestimatedaward.Thus,inasense,incorporatingprior-awardinformationas

guidanceindeterminingadamagesawardallowsthetrieroffact(implicitly)tostrikea

balancebetweenbiasandvariancesoastominimizeerror.Byacceptingthepossibilityof

introducingsomebiasduetomaterialdifferencesbetweenthepresentcaseandprior

cases,itispossibletogainfarmore,intermsofreliability,duetothereductioninaward

variabilitythatfollowsfromsuchguidance.

Furthermore,theincorporationofprior-awardinformationshouldnotbeviewedas

sufferingfromcircularity—thatis,asamethodthatisdefeatedbyitsrelianceon

66Seeid.(discussing“judgmentvariability”and“claimvariability”);seealsoHillelJ.Bavli,SamplingandReliabilityinClassActionLitigation,2016CARDOZOL.REV.DENOVO207,210–12(2016).

94

informationthatentailsthesameflawsthatthemethodisitselfattemptingtoaddress.

Rather,suchmethodsshouldbeunderstoodasfacilitatingtheimprovementofawardsby

allowingforinformationsharingacrosscases.Intheframeworkdescribed,forexample,an

idealmethodforreducingerrorfromvariabilitywouldgenerateadamagesawardby

repeatedlyadjudicatingacasemanytimesindependentlyandusingthemeanofthe

repeatedadjudicationsastheultimateaward.Butsuchmethodsaregenerallyimpractical.

Methodsinvolvingtheincorporationofprior-awardinformationcanbeunderstoodas

efficientalternatives.Suchmethodsaimtocontrolvariabilitythroughinformation-sharing

acrossdifferentcases,ratherthanacrossreplicationsofthesamecase.Thecostofefficiently

reducingjudgmentvariability,however,isthepotentialforbiasarisingfromclaim

variability.

Inordertogainabetterunderstandingoftheeffectsofprior-awardinformationon

error,letusconsidertwoextremescenarios.First,supposethattheprior-award

informationis“dogmatic”inthesensethatitentirelydominatesthedamagesaward

withoutleavingroomforanyinfluencefromthefactsofthepresentcaseitself—for

example,supposethejurydecidesdogmaticallysimplytoapplytheaverageoftheprevious

awardsasitsawarddetermination,withoutconsiderationofthefactsofthecurrentcase.

Inthiscase,therewouldbenoerrorfromvariability,sincetheaverageoftheprevious

damageawardsisconstant;buttotheextentthattheaverageofthepriorawardsis

differentfromthecorrectcurrentawardα,therewouldbesubstantialerrorfrombias.On

theotherhand,supposethattheprior-awardinformationhasnoinfluenceonthecurrent

damagesaward—forexample,supposethejurydecidesdogmaticallythatitwillarriveat

95

anawarddeterminationbasedonthefactsofthecasealone,withoutinfluencefromprior

damageawards.Inthiscase,assumingjurydamagedeterminationsareinitially(i.e.,

withoutinfluencefrompriorawards)unbiased—thatis,theyachievethecorrectawardα

onaverage—thentherewouldbenoerrorfrombias,butsubstantialerrorfromvariability.

CCGimprovesthereliabilityofawardsforpainandsufferingandpunitive

damages—awardtypesthatsufferfromparticularlyhighdegreesofvariability—by

facilitatingabalancebetweenminimizingvariabilityandintroducingthepossibilityofbias.

Thequalityofthebalanceisdeterminedbythestrengthofthemethodforidentifying

comparablecasesanddistillinginformationforconsiderationbythetrieroffact.

Tobesure,letusconsidertwoimportantissues.First,assumingan“appropriate”

setofpriorcases,whatistheideallevelofinfluencethatshouldbeaffordedtoprior-award

information?Second,howcanwebecertainthatthepriorcasesidentifiedaresufficiently

comparabletothepresentcasesoastoimprovereliability?

Thefirstissue—theidealinfluenceofprior-awardinformation,assumingan

appropriatesetofpriorcases—dependsonthejudgmentvariability(σ2)oftheawardin

thepresentcaseandtheclaimvariability(τ2)ofthepriorcases.Higherjudgment

variabilitysuggestsweakerinformationobtainedfromthepresentcaseandgreater

influenceofthepriorawards;higherclaimvariabilitysuggestsweakerinformation

obtainedfromthepriorcasesandgreaterinfluenceofthepresentcase.

Forexample,consideratortcaseinwhichtheevidenceagainstthedefendantis

overwhelmingandthedamagesincurredbytheplaintiffareclear.Assumethatthe

evidenceissufficientlystrongthat,ifthecasewerelitigatedtentimesindependently,the

96

damageawardsinalltenreplicationswouldbeapproximatelythesame—say,$100,000.In

thiscase,theevidenceissogreatandthejudgmentvariabilitysolow(almostzero)that

thereislittletobegainedfromintroducingprior-awardinformationandthepossibilityof

bias.Ontheotherhand,consideracaseinwhich,althoughtheremaybestrongevidence

regardingthefactsofthecase,thereisonlyweakevidenceregardingtheappropriate

damagesaward—asisalmostalwaysthecaseforawardsforpainandsufferingand

punitivedamages.Ifthecasewererepeatedlyadjudicatedtentimesindependently,the

damageawardsacrossthetenreplicationswouldvarywildly.Inthiscase,thedamages

awardishighlyvariableandthereismuchtobegainedbyintroducingprior-award

information.Further,theprior-awardinformationisespeciallybeneficialifclaim

variabilityislowandpriorawardsprovideclearinformation.Forexample,assumethatthe

factsandlegalissuesinthepresentcaseareclear,andthatthecourtconfidentlyidentifies

asetoftencomparablecasesthatinvolvesimilarfacts—forexample,similarinjuries

arisingfromsimilarcircumstances.Assumefurtherthatthedamagesawardedintheten

casesareverysimilar—say,approximately$50,000.Inthiscase,inwhichjudgment

variabilityishighandclaimvariabilityislow,prior-awardinformationshouldhaveahigh

degreeofinfluence.67

67Moreformally,itcanbeshownthat,if,asabove,thedamagesaward𝛼isdistributedwithmeanαandvarianceσ2,andthe“correct”damagesawardαisdistributedwithmeanµandvarianceτ2(thus,ifwehavejudgment variabilityσ2and claimvariabilityτ2), then theoptimal damages awardweights current caseinformationby1/σ2andprior-awardinformationby1/τ2.Inotherwords,theidealinfluenceofthepriorawards(andparticularlythemeanofthepriorawards)isinverselyproportionaltotheclaimvariability(τ2) of the prior awards relative to the judgment variability (σ2) of the award in the present case. Seegenerally Efron & Morris, supra note 59; W. James & Charles Stein, Estimation with Quadratic Loss, 1PROCEEDINGSOFTHEFOURTHBERKELEYSYMPOSIUM361(1961).Notethatsincewecannot,inpractice,knowthevalueofµ,themeanofthecorrectawards,weinsteadestimateµusingthemeanofthepriorawards,and

97

Itisworthreemphasizingherethatitisincorrecttoconclude,asnumerous

commentatorshave,that,becausepriorawardsmaysufferfromthesamearbitrariness

thatweaimtoaddressinthepresentcase,CCGwouldbeineffectiveinreducing(or,worse,

wouldcompound)awardvariability.Evenassumingthe(not-unlikely)scenariothat

judgmentvariabilityisapproximatelyequalacrosscases(asIhaveassumedabove),the

foregoinganalysismakesclearthattheidealinfluenceofprior-awardinformationcan

neverthelessbeveryhigh.Inotherwords,CCGmayneverthelesssignificantlyreduce

variabilityandimprovereliability.Thismakessense,sinceCCGallowsforthesharingof

informationacrossnumerous(althoughvariable)cases;itprovidessignificantguidance

whererelevantinformationisotherwisescarce.

Consideranextremeexampleinwhichjudgmentvariabilityisextremelyhighforall

cases(thepriorcasesaswellasthepresentcase)andclaimvariabilityisextremelylow.

Here,althoughpriorawardssufferfromthesameextremearbitrarinessthatweaimto

addressinthepresentaward,acourtwouldbenefitimmenselyfromCCGmethods.

Specifically,becauseclaimvariabilityisextremelylow,theprior-awardinformation

obtainedherewouldbeapproximatelyequivalenttoprior-awardinformationthatcouldbe

obtainedfromthe“ideal”proceduredescribedabove,wherebyadamagesawardis

calculatedbyaveragingoverawardsobtainedfromrepeatedindependentadjudications

(withdifferentjuries,judges,lawyers,presentationsoftheevidence,etc.)ofasinglecase,

“shrink” toward this value. We can estimate the variance terms as well. Notationally, the “shrinkage

estimator”canbewrittenas𝛼U =ij-G kl-

mj-G ml-.

98

thecaseathand.Theawardsobtainedfromrepeatedadjudicationswouldvary

considerably,sincejudgmentvariabilityisextremelyhigh.Buttheultimateaward,

calculatedbyaveragingalloftheawardsobtainedfromtherepeatedadjudications,is

extremelyprecise—itissubjecttoalmostnovariability.

Assuggestedabove,methodsinvolvingrepeatedadjudicationareidealbutcostly.

CCGmethodsareefficientalternativesthatcomeatthecostofpossiblebiasthatmayarise

dueto“misalignment”—wherethemeanofthecorrectawardsinthepriorcasesis

differentfromthecorrectawardinthesubjectcase—andperhapsduetoclaimvariability,

orheterogeneityinthesetofpriorcases.68Inourcurrentexample,however,biasisnota

problem,becausethereisnomisalignmentproblemandclaimvariabilityisassumedtobe

extremelylow.Thus,althoughthepriorawardsaresubjecttoextremelyhighjudgment

variability,intheaggregatetheyallowforanextremelyreliableoutcomethroughthe

sharingofinformationacrosscases.Furthermore,wecanrelaxtheassumptionof

extremelylowclaimvariability.Higherclaimvariabilitymaycausebias;buttheharmto

reliabilityassociatedwithanyintroductionofbiasisoftenwelloutweighedbythe

reliabilitybenefitsassociatedwiththereductioninjudgmentvariability.Inanycase,the

introductionofclaimvariabilitydoesnotaffecttheconclusionthatCCGmethodsare

effectiveinreducingjudgmentvariabilityandimprovingreliabilitynotwithstandingthe

potentialarbitrarinessassociatedwiththepriorawards.

68Seeinfranote81.

99

Thesecondissue—thereliabilitybenefitsofCCGinlightoftheactualcomparability

ofthepriorawards—dependsonwhetherthepotentialforbiasisoutweighedbythe

benefitsofreducingvariability.Thisisavalidconcerninthesensethatthereliability

benefitsofCCGareindeeddependentonthematerialcomparabilityofthepriorcasesto

thepresentcase.

Fortworeasons,however,thelikelihoodofidentifyingasetofcasessufficiently

inappropriate,relativetothepresentcase,toharmreliabilityisverylow.First,because

awardsforpainandsufferingandpunitivedamagesarehighlyvariable,thepotential

reliabilitybenefitsofCCGaregreat.Asdiscussedabove,thehigherthevariabilityofthe

award,thegreaterthepotentialbenefitsofCCG,andthegreaterourtoleranceforbias.The

potentialforharmfrombiasarisingfrommisalignmentwouldbemoreconcerning,for

example,ifacourtwereinterestedinapplyingprior-awardinformationtoreducethe

already-lowvariabilityofanawardforacertaintypeofeconomicdamages.However,the

highvariabilityofawardsforpainandsufferingandpunitivedamages—sufficientlyhighto

promptdrastic(andhighly-biasing)measuressuchasdamagecaps—suggeststhatthe

benefitswithrespecttovariabilityarelikelytodominateanybiasarisingfrom

misalignment(oranynegativeeffectsarisingfromhighclaimvariability).

Second,thebiasintroducedbyCCGmethodsislikelytobelow.Statistically,the

appropriatenessofasetof“comparable”cases,relativetothepresentcase,involvesthree

majorfactors:1)the“alignment”ofthemeanofthe“correct”awardsinthepriorcases

withthe“correct”awardinthepresentcase,2)thevariabilityofthecorrectawardsinthe

priorcases,and3)thenumberofpriorcases,orthe“samplesize.”Inpractice,acourt

100

shouldbeconcernedwiththealignmentofthematerialfactsofthepriorcaseswiththose

ofthepresentcase,thesubstantivebreadthofthepriorcases,andthesamplesize.

Notethatwecannotperfectlypredicthowatrieroffactwillinfactincorporate

guidancefrompriorawards.Forpurposesofthisdiscussion,andfortheguidanceIprovide

inSection5belowregardingthedevelopmentofcomparable-caseinformation,Iassume

thatthetrieroffactwillnotact“irrationally”byaffordingmoreinfluencetohighly-

dispersedpriorawardsthantominimally-dispersedpriorawards.Itisnotnecessaryfor

thetrieroffacttobehaveperfectlyrationally,buttheforegoingassumptionprovidesa

goodstartingpointforguidance.We“trust”thetrieroffacttoactapproximatelyrationally

innumerouslegalcontexts;andindevelopingguidanceforthetrieroffact,thecourt

arguablyshouldassumesomedegreeofrationality,asitdoesinprovidingotherformsof

guidance.

Statistically(usingfundamentalsof“shrinkageestimation”),incorporatingprior-

awardinformationwillgenerallyimprovereliability,aslongasthepriorawardsarenot

tightlybound(i.e.,withlowvariability)aroundameanthatisfarfromthecorrectawardin

thepresentcase69—inpractice,aslongasthepriorawardsarenottightlyclusteredaround

anawardthatreflectsfactsmateriallydissimilartothepresentcase.

Highprior-awardvariabilityitselfis(absentextremeconditions)unlikelytoharm

thedamagesaward,sincethevariabilityofawardsforpainandsufferingandpunitive

damagesishigh,and,assuggestedbythediscussionabove,theweightaffordedtheprior-

69SeegenerallyEfron&Morris,supranote59;James&Stein,supranote67.

101

awardinformation(and,statistically,themeanofthepriorawardsinparticular)is

proportionaltotheinverseofthevariabilityofthepriorawards.Thismeansthathighly

dispersedpriorawards(e.g.,priorawardsthatspanawiderangeofvalues)willhavelittle

influenceontheawarddetermination(inthemodelabove,𝛼Uwillbeapproximatelyequal

to𝛼).

Additionally,althoughuncomparablepriorcasescancausesignificantbias,this

concernfadeswhenweconsidertherelativelyremoteconditionsnecessaryforCCGto

harmreliability,inthesenseofincreasingawarderror.Inparticular,forCCGtoharm

reliability,itwouldgenerallyrequireacombinationofsignificantlydissimilarcasesand

lowprior-awardvariability—acombinationthatishighlyunlikelyifthepriorcasesare

specificallyselectedtobecomparabletothepresentcase,andwherethevariabilityofthe

awardscanoftenreflect(andperhapsshouldreflect)thecourt’slevelofconfidence

regardingthecomparabilityofthepriorcasestothepresentcase.70

Inshort,wecanexpectthatareasonablemethodforidentifyingprior“comparable”

caseswillresultinprior-awardinformationthatislikelytoimprovereliability.71Ofcourse,

themorecomparablethepriorcasesaretothepresentcase,thebettertheguidanceforthe

trieroffact,andthegreaterthereliabilitybenefits.Inthefollowingsection,Idiscussin

greaterdetailthedevelopmentofcomparable-caseinformation.

70SeeSection5foradiscussionofmethodsforselectingcomparablecases.

71A formalderivationof theparticularmeanandvariabilityconditions thatwouldcauseareduction inreliabilityisbeyondthescopeofthecurrentarticle.

102

5. DevelopingComparable-CaseInformation

Inthissection,Idiscussanumberofconsiderationsforestablishingcomparable-

case(or“prior-award”)information.Iaddresstwoareasofconcern:1)identifyingasetof

materially“comparable”cases,and2)distillingcomparable-caseinformationfromthese

casesforconsiderationbythetrieroffact.

5.1 Identifying“Comparable”Cases

TouseCCG,acourtmustidentifyasetofcomparablecases.Assuggestedabove,a

courtshouldbeconcernedwiththreefactors:thealignmentofthepriorcaseswiththe

subjectcase,thebreadthofthepriorcases,andthenumberofpriorcases.Thesuitabilityof

thecasesandexpectedbenefitsoftheguidancedependonthecourtscareinbalancing

thesefactors.Forexample,priorawardswithlowerbreadthgenerallyproduce,inasense,

more“specific,”andthereforemoreinfluential,guidance;butlowerbreadthalsoresultsin

asmallersamplesize,whereasalargersampleprovidesmoreinformationandtherefore

betterguidance.Additionally,thealignmentofthepriorcasesaffectsthebiasintroduced

bythepriorawards,andthebreadthofthepriorcases,whichaffectstheweightor

influenceofthepriorawards,mayreflectthecourt’sconfidenceinthealignmentofthe

casesidentified.

Intermsofalignmentandbreadth,themostsignificantfactoraffectingacourt’s

identificationofpriorcasesislikelyitsunderstandingofthesubjectcaseandwhichfacts

andissuesarefundamentaltoanappropriatecomparison.

Consider,forexample,JudgePosner’sanalysisinJutzi-Johnson.Thatcaseinvolved

allegationsthatajailwasnegligentinfailingtoidentifyaprisonerasasuicideriskandtake

103

precautionarymeasures,resultingintheprisonerhanginghimself.Commentingonthe

parties’identificationofcomparisoncases,JudgePosnerexplained:

Theplaintiffcitesthreecasesinwhichdamagesforpainandsufferingrangingfrom $600,000 to $1million were awarded, but in each one the pain andsufferingcontinuedforhours,notminutes.Thedefendantconfineditssearchfor comparable cases tootherprisonsuicidecases, implying thatprisonersexperiencepainandsufferingdifferentlyfromotherpersons,sothatitmakesmoresensetocompareJohnson’spainandsufferingtothatofaprisonerwhosuffered a toothache than to that of a free personwhowas strangled, andconcludingabsurdly thatanyaward forpainandsuffering in this case thatexceeded $5,000 would be excessive. The parties should have looked atawardsinothercasesinvolvingasphyxiation,forexamplecasesofdrowning,whicharenumerous....Hadtheydoneso,theywouldhavecomeupwithanawardintherangeof$15,000to$150,000.72

JudgePosnerthusdisagreedwiththealignmentoftheplaintiff’scases,which

involvedpainandsufferingthatcontinuedforhoursratherthanminutes.Healsodisagreed

withthealignment,aswellasthebreadth,ofthedefendant’scases,whichwere“confined”

to“otherprisonsuicidecases”—asthough“prisonersexperiencepainandsuffering

differentlyfromotherpersons”—leadingto“absurd”conclusionsregardingasuitable

award.73Finally,JudgePosnerdisplayedconcernforsamplesize,suggestingthatwidening

thebreadthofthepriorcasestothoseinvolvingasphyxiationwouldhaveresultedin

“numerous”priorcasesforcomparison.74Samplesizeisimportantbecausepriorawards

aresubjecttojudgmentvariability:ifacourtselectsonlyasinglecaseforcomparison,then,

evenifthecourtisextremelyconfidentthatthepriorcaseisindeedcomparable,itwould

remainunclearwhethertheawardinthatcaseisduetothespecificfactsofthecaseorto

72Jutzi-Johnsonv.UnitedStates,263F.3d753,760-61(7thCir.2001).

73Id.

74Id.

104

randomness.Acourtmaybejustifiedinsacrificingsomedegreeof“comparability”for

purposesofincreasingthesamplesize.Ofcourse,inpractice,acourtmaydeterminethat

thematerialfactsofacasearetoouniqueforcomparison,andmayaccordinglyforegoany

applicationofCCG.

Withthesefactorsinmind,thecourtshouldarriveataprocessforselectinga

suitablesetofpriorcases.Selectionmethodsshouldremainflexiblesothatcourtsmay

catersuchmethodstotheuniquecircumstancesofthecaseathand.Initially,courtscan

obtainguidancefromcomparativeapproachesusedregularlytodeterminedamagesin

variouscivilcontexts.75

Thus,considerasimpleprocedureinwhichthecourtaskseachpartytoidentifya

setof“comparable”cases.Suchaprocedureislikelytoresultinasetofextreme(although

perhapsunbiased)cases,sinceeachpartyselectscasestomaximizeitsowninterests.76On

theotherhand,amethodbywhichthecourt,ratherthantheparties,selectsasetof

“comparable”casesislikelytoproducelessvariability,butissusceptibletoanumberof

drawbacks,includingthepotentialforjudicialbiasandundueburdenonthecourt.

Alternatively,thecourtmaydirectthepartiestoagreeonasetofpriorcases;butthismay

bedifficultandinefficientwithoutadditionalstructurefromthecourt.77

75 See infra Section 6. Courts may also obtain guidance from the class action context, where courtssometimesselectasetof“representative”or“bellwether”casesforsettlementorlitigationpurposes.SeegenerallyBavli,supranote50.

76SeegenerallyLorenH.Brownetal.,BellwetherTrialSelectioninMulti-DistrictLitigation:EmpiricalEvidenceinFavorofRandomSelection,47AKRONL.REV.663,674-75(2014).

77Seegenerallyid.at675.

105

Althoughthecircumstancesofaparticularcasemaycallforoneofthemethods

above,aparticularlyusefulapproachisoneinwhichthecourtselectsafinalsetofcases

fromapoolidentifiedbythelitigants.Variationsofthismethodhavebeenusedinvarious

civilcontexts.78Itallowsforsignificantflexibility,andbenefitsfrombothalitigant-based

selectionprocessandtherelativeobjectivityofthejudge.

Finally,itispossibletoinclude(implicitly)thetrieroffactintheselectionprocess

byprovidingitwithqualitativeinformationregardingthesetofcomparablecases,thus

allowingittoweighttheprior-awardinformationbased,inpart,onitsassessment

regardingthealignmentofthepriorcases.Idiscussthispossibilityfurtherinthefollowing

subsection.

5.2 DistillingComparable-CaseInformation

Oncethecourthasselectedasuitablesetofpriorcases,itmustdecidewhat

informationshouldbeconsideredbythetrieroffactasguidance.AstudybyProfessor

MichaelSaksetal.examinedtheeffectsofvariousformsofjuryguidanceonamounts

awardedforpainandsufferinginpersonalinjurycases.79Theyfoundthat,whiledamage

capsperformedpoorlyintermsofvariabilityanddistortionofawardmagnitude,various

formsofdistributionalinformation—inparticular,an“awardinterval,”an“awardaverage-

plus-interval,”and“awardexamples”—wereeffectiveinreducingvariabilitywhile

78See infraSection6; see alsoBrownet al., supranote76, at 672-76 (describing cases employing suchproceduresinthebellwethercontext).SeegenerallyAlexandraD.Lahav,TheCasefor“TrialbyFormula”,90TEXASL.REV.571(2012).

79SeeSaksetal.,supranote15.

106

distortingawardmagnitudeminimallyornotatall.80Theseresultssupportthemodel

describedinSection4.Inparticular,inlinewiththeanalysisabove,prior-award

informationisexpectedtocauseareductioninvariabilityandminimal(ifany)distortionof

awardmagnitudewherethedistributionofprior-awardsissimilartothedistributionthat

wouldbeobtainedintheabsenceofprior-awardinformation.Indeed,Saksetal.obtained

thevaluesoftheguidanceprovidedtosurveyparticipantsfromtheresultsoftheirpilot

study.Specifically,theyusedthemedianofthepilotstudyfortheawardaverage,the10th

and90thpercentilesfortheawardinterval,the10th,50th,and90thpercentilesforthe

awardaverage-plus-interval,andthe10th,35th,65th,and90thpercentilesfortheaward

examples.81

ThediscussioninSection4suggeststhatcourtsshouldchooseinformationforms

thatprovidejurorswithinformationregardingthe“center,”aswellasthe“spread,”or

variability,ofthedistributionofthepriorawards.82Theawardinterval,awardaverage-

plus-interval,andawardexampleshavetheadvantage,relativetoproviding,forexample,

onlytheawardaverage,thatjurorsareprovidedinformationregardingthevariabilityof

thepriorawards.Thisenablesthemtoweighttheprior-awardinformationappropriately.

Theawardexamplesformisparticularlyappealing,becauseitprovidesjurorswiththe

80Seeid.at249-55.Saksetal.observedmixedresultsforthe“awardaverage”form.Seeid.at253-55.

81Id.at248.Theauthors’resultsforthecapcondition,whichcausedadistortioninthesize,ormagnitude,oftheawardandmixedeffectsonvariability(dependingontheseveritycategory)canbeinterpretedassupportiveofthemodelherein,butmayalsohaveimplicationsfortheeffectsofprior-awardinformation.

82Itispossiblethat,empirically,jurorswillnotinfactweighttheprior-awardinformationaccordingtotherelative variability of such information; butwithout significant evidence to the contrary, courts shouldarguablyprovidejurorswithrelevantvariabilityinformation.

107

mostinformation,relativetotheotherinformationforms,regardingprior-award

variability.

Further,inadditiontonumericalprior-awardinformation,itmaybebeneficialfor

thecourttoallowthetrieroffacttohearqualitativesummaryinformationregardingthe

priorcases.Providingthetrieroffactwithsuchinformationwouldstrengthenitsrolein

thepossiblyfact-intensiveidentificationprocess,andwouldmitigatetheeffectsof

potentialbiasfrommisalignment.Assuggestedabove,providingthetrieroffactwith

qualitativecaseinformationimplicitlyincludesitinthecase-selectionprocessbyenabling

ittoweighttheprior-awardinformationbased,inpart,onitsassessmentofthe

comparabilityofthepriorcases.83

Ifthecourtallowsthetrieroffacttohearqualitativecaseinformation,itmust

determineanappropriateformforsuchinformation—forexample,abriefdescription

regardingthesetofcasesasawhole,orasetofshortfactualsummariesregardingeach

individualcase.

AsdescribedinSection6below,thereissubstantialprecedentforallowingajuryto

hearcomparable-caseinformationintheformofevidenceintroducedasexperttestimony.

Ajurymaythereby“consider...argumentsabouttheeffectof[purporteddifferences]on

thevalidityof[the]comparisonand...adjustitsdamageawardaccordingly.”84

83Thecourtshouldbalancethebenefitsofallowingqualitativecaseinformationwiththecosts(includingattorneys’fees,delays,etc.)ofallowingsuchinformation.

84SyufyEnterprisesv.AmericanMulticinema,Inc.,793F.2d990,1003(9thCir.1986).

108

6. Conclusion

Intheprevioussections,IaddressedthreemajorobjectionstoCCGmethods.First,I

explainedthatCCGcanreduceunpredictabilityandimprovethereliabilityofawards—

notwithstandingthejudgementvariabilityassociatedwiththepriorawards—byallowing

forthesharingofinformationacrosscases.Second,Idiscussedhowinformationregarding

awardsincomparablecasescanbeunderstood,notasanalternativetoassessingdamages

forpainandsufferingoranappropriatepunitivedamagesaward,butasfundamentalto

assessingsuchvalues.Finally,Iexplainedthatitisnotnecessaryforthepriorcasestobe

perfectlycomparabletothesubjectcase,andthat,assumingareasonablemethodfor

identifying“comparable”cases,itisunlikely—particularlyforawardsforpainand

sufferingandpunitivedamages—thattheywilldiminish,ratherthanimprove,reliability.

Moreover,itisimportanttounderstandthatusingcomparable-caseinformationas

guidanceindeterminingawardsforpainandsufferingandpunitivedamagesisnota

remoteconceptthatrequiresdramaticchangetocurrentpractices.Indeed,courtshave

recognizedtheimportanceofcomparable-caseinformationasguidanceindetermining

awardsinarangeofcontexts.

First,thereissubstantialprecedentfromthejudicial-reviewcontextforallowing

informationregardingpriorawardstoinfluencepresentawardsand,inparticular,forthe

useofcomparable-caseinformationasguidanceindetermininganappropriateawardor

rangeofawards.Indeed,courtshavedescribedthecomparisonofawardstopriorawards

forsimilarinjuriesas“[a]mainstayoftheexcessivenessdetermination,”notingthat“[t]his

useofcomparisonisarecognitionthattheevaluationofemotionaldamagesisnotreadily

109

susceptibleto‘rationalanalysis.’”85Forexample,courtsinNewYorkhaveheldthat,in

determiningwhetheranawardisexcessive,areviewingcourtmustconsiderawardsin

comparablecases86—thatthe“determinationofwhetheracompensatorydamagesaward

isexcessiveshouldnotbeconductedinavacuum,butinsteadshouldincludeconsideration

oftheamountsawardedinother,comparablecases.”87TheU.S.SupremeCourthasitself

indicated,inthecontextofdeterminingwhetheraparticularpunitivedamagesaward

exceededtheamountpermittedbytheDueProcessClauseoftheConstitution,that

“[c]omparingthepunitivedamagesawardandthecivilorcriminalpenaltiesthatcouldbe

imposedforcomparablemisconductprovidesa[n]...indiciumofexcessiveness.”88

Second,thereissubstantialprecedentfromthebench-trialcontextforCCG—and

specifically,forconsiderationofcomparable-caseinformationbythetrieroffactfor

guidanceindeterminingawardsforpainandsufferingandpunitivedamages.Infact,

85Salinasv.O’Neill,286F.3d827,830-31(5thCir.2002)(internalcitationsomitted).

86Allav.Verkay,979F.Supp.2d349,372(E.D.N.Y.2013)(emphasisadded).

87DiSorbov.Hoy,343F.3d172,183(2dCir.2003)(internalquotationmarksandcitationsomitted).SeeGeressyv.DigitalEquip.Corp.,980F.Supp.640,656(E.D.N.Y.1997)aff’dinpartsubnom.Maddenv.DigitalEquip.Corp.,152F.3d919(2dCir.1998)(“CPLR5501(c)’sconceptionofreasonablecompensationcannotexistinavacuum.Thereneedstobesomepointofreference.Witheconomicdamages,thecourtmayrelyon traditional methods of economic analysis. As for the non-economic pain and suffering award, thereviewingcourtmustbeginbyidentifyingsomegroupofsimilarcasestoserveasareferent.Thistaskisdifficult, particularly in cases exploring relativelynew types of injuries and claims such as those in theinstantcaseinvolving[repetitivestressinjury(RSI)]claimsagainstakeyboardmanufacturer.Caseswithsimilar causal agents, similarly-nameddiagnoses, or similar reductions in quality of lifemight serve asbenchmarks.”).

88SeeBMWofN.Am.,Inc.v.Gore,517U.S.559,583(1996).SeealsoDegorskiv.Wilson,No.04CV3367,2014WL 3511220, at *1 (N.D. Ill. July 16, 2014) (“In assessing whether a punitive damage award isconstitutionally appropriate, the Supreme Court has directed courts to focus their evaluation on threeguideposts:(1)thereprehensibilityofthedefendant’sconduct;(2)therelationshipbetweentheamountof thepunitivedamages awardedand theharmorpotential harmsufferedby thePlaintiff; and (3) thedifference between the punitive damages award and the civil penalties authorized or imposed incomparablecases.”).

110

considerationofcomparable-caseinformationisoftenexpected.Forexample,inJutzi-

Johnson,JudgePosnerremarkedthatmostcourts“treatthedeterminationofhowmuch

damagesforpainandsufferingtoawardasastandardless,unguidedexerciseofdiscretion

bythetrieroffact,reviewableforabuseofdiscretionpursuanttonostandardtoguidethe

reviewingcourteither.”89Headvised:

Tominimize the arbitrary variance in awards bound to result from such athrow-up-the-handsapproach,thetrieroffactshould...beinformedoftheamountsofpainandsufferingdamagesawardedinsimilarcases.Andwhenthe trier of fact is a judge, he should be required as part of hisRule 52(a)obligationtosetforthinhisopinionthedamagesawardsthatheconsideredcomparable. We make such comparisons routinely in reviewing pain andsufferingawards,asdoothercourts.Itwouldbeawisepracticetofollowatthetriallevelaswell.90

Third,thereisprecedentfromvariouscivilcontextsforaskingjurorstomake

comparisonsforpurposesofdeterminingdamages.Thecomparisonsoftenrelatetogoods

orservicesforwhichthereisaneconomicmarket,andthecomparisonsareundertakenfor

purposesofprovidinginformationregardingthemarket,which,inturn,provides

informationregardingtheappropriatedamagesaward.

Forexample,inthecontextofdetermining“justcompensation”ineminentdomain

litigation,itiscommonforacourttoaskthejurytoundertakea“comparablesales”

89Jutzi-Johnsonv.UnitedStates,263F.3d753,759(7thCir.2001).

90Id.;seealsoArpinv.UnitedStates,521F.3d769,776-77(7thCir.2008)(explainingthatthelowercourt“shouldhaveconsideredawardsinsimilarcases,bothinIllinoisandelsewhere,”andthat“[c]ourtsmaybeabletoderiveguidanceforcalculatingdamagesforlossofconsortiumfromtheapproachthattheSupremeCourt has taken in recent years to the related question of assessing the constitutionality of punitivedamages”);Zurbav.UnitedStates,247F.Supp.2d951,961(N.D. Ill.2001)aff’d,318F.3d736(7thCir.2003)(consideringawardsin“comparable”cases);Elkv.UnitedStates,87Fed.Cl.70,97(2009)(“[v]ariouscases,includingthosearisingundertheFTCA,suggestthatjudgesfacedwiththeprospectofdeterminingpainandsufferingawardsshouldlooktotheawardsinsimilarcases”);Cochranv.A/HBatteryAssociates,909F.Supp.911,917(S.D.N.Y.1995)(“courtshavelongrevieweddamageawardsinothersimilarcasestodeterminethescopeofreasonablenessinaparticularcase”).

111

analysis,forwhichthejuryhearstestimonyregardingthesellingpriceofcomparable

properties.91Insuchcases,thecourtintendstoprovidethejurywithinformation

regardingthemarkettowhichthepropertyatissuebelongs,andtherebyfacilitatethe

jury’sassessmentofjustcompensationandthusanappropriatecompensatorydamages

award.Indeed,comparablesalesevidenceisoftenviewedasthe“bestevidence”ofmarket

value.92Similarly,damageawardsforbreachesofcontractforthesaleofgoodsorservices

areoftencomputedbylookingatthemarketpriceofcomparablegoodsorservices.93And

courtsalsorelyonthejurytomakecomparisonsinarrivingatdamagescalculationsinthe

antitrustcontext.94

91SeeUnitedStatesv.1,129.75AcresofLand,MoreorLess,inCross&PoinsettCtys.,473F.2d996,998(8thCir. 1973) (citing cases, and holding that, “[g]enerally, evidence of sales of comparable property ispersuasiveevidenceofmarketvalue,eitherasdirectprooforinsupportofawitness’sopinion”).

92 SeeUnited States v. 24.48Acres of Land, 812 F.2d 216, 218 (5th Cir. 1987) (“First,we reiterate theprinciplethatthebestevidenceofmarketvalueiscomparablesales”)(citingUnitedStatesv.8.41Acres,680F.2d 388, 395 (5th Cir. 1982)). See also John J. Dvorske & Ann K.Wooster, Condemnation of Property,ComparableSales,7FED.PROC.,L.ED.§14:90(December2015).

93U.C.C.§2-713(“themeasureofdamagesfornon-deliveryorrepudiationbytheseller[ofgoods]isthedifferencebetweenthemarketpriceatthetimewhenthebuyerlearnedofthebreachandthecontractpricetogetherwithanyincidentalandconsequentialdamagesprovidedinth[e]Article(Section2-715),butlessexpensessavedinconsequenceoftheseller’sbreach”).SeealsoGulfPowerCo.v.CoalsalesII,LLC,522F.Appx.699,707(11thCir.2013)(holding,inacaseinvolvingabreachofacontractforthepurchaseandsaleofcoal,that“theappropriatemeasureofdamageswouldbethemarketremedyprovidedforin§672.713[oftheU.C.C.],inwhichdamagesarecalculatedbycomparingthecontractpricetothefairmarketvalueofcomparablegoodsontheopenmarket”).

94Forexample,inSyufyEnterprisesv.AmericanMulticinema,Inc.,793F.2d990(9thCir.1986),counterclaimantAMCbaseditsdamagescalculationoncomparisonswithpurportedlycomparablemarkets.Itpresentedsuchcomparisonstothejury,whichultimatelyrenderedanawardaccordingly.Seeid.at1002-03.TheNinthCircuitrejectedtheargumentthatdamagesmaynotbecomputedbasedoncomparablemarkets.Seeid.ThecourtthenrejectedtheargumentthatthecomparisonmarketswerenotcomparabletothemarketatissueandtherebyinappropriatelyinflatingAMC’sdamagescalculation.Seeid.at1003.Thecourtheldthat“[c]omparabilityisaquestionoffact”andthat“[i]twasforthejurytoconsider[theopposingparty’s]argumentsabouttheeffectofthe[claimeddifferences]onthevalidityofcomparisonandtoadjustitsdamageawardaccordingly.TherewassufficientevidencetoallowajurytorenderadamageawardforAMConthebasisofthecomparison....”Id.

112

AdetailedexaminationofthelegalfoundationsofCCGisbeyondthescopeofthis

article.Thelegalcontextaboveisintendedsimplytoshowthatthereisalonghistory,

acrossvariouscontexts,ofusingcomparable-caseinformationasguidanceindetermining

damages.Indeed,courtshaveappliedsuchmethodsnotwithstandingtheobjections

addressedinthisarticle.Forexample,considertheobjectionthatcourtswillcompoundthe

arbitrarinessofawardsbyobtainingguidancefrompriorawardsthatpresumablysuffer

fromthesamearbitrarinessthatcourtshopetoaddress.Thisobjectionappliessimilarlyto

comparable-caseinformationconsideredinthebench-trialcontext,butbench-trialjudges

neverthelessconsidercomparable-caseinformation.Additionally,theobjectionthat

methodsinvolvingtheuseofpriorawards“failtoaddressthefundamentalissueofhow

oneshouldinitiallyassessthevalueofpain-and-sufferingdamages”95orarriveatan

appropriatepunitivedamagesawardappliessimilarlytotheuseofcomparable-case

informationinthejudicial-reviewandbench-trialcontexts;butsuchinformationisstill

usedinthesecontexts.Finally,theobjectionthatthevalidityofmethodsinvolving

comparable-caseinformationreliesheavilyonthepresumptionthatthepriorcases

identifiedareindeedmateriallycomparabletothepresentcaseappliessimilarlytoallof

thecontextsdiscussedabove;96butsuchmethodsareneverthelessused.

MyaiminthisarticlewastoaddressthemajorobjectionstoCCGmethodsby

explainingsimply,butformally,whatmanycourtsandcommentatorshaverecognized

95Avraham,supranote19,at104.

96Theobjectionappliesdirectlytothejudicial-reviewandbench-trialcontextsandanalogouslytothemarket-comparisonscontext.

113

implicitly:thatCCGiseffectiveinreducingunpredictabilityandimprovingthereliabilityof

awardsforpainandsufferingandpunitivedamagesbyallowingforthesharingof

informationacrosscases.

114

IV. ShrinkageEstimationintheAdjudicationofCivilDamageClaims*

(Coauthor:YangChen)

1. Introduction

Alegalproceedingcanbeunderstoodasaprocedureforgeneratinganoutcomethat

servesasanestimateofthe“correct”outcomeassociatedwithalegalclaim.Inthissense,a

criterionformeasuringthestrengthofalegalprocedureisreliability:thedegreetowhich

theprocedurecanbeexpectedtogenerate“accurate”outcomes,outcomesthatareclosein

proximitytothe“correct”outcome.1Intworecentarticleswrittenbyoneoftheauthors,it

isarguedthatcertainclaimaggregationmethods,methodsinwhichtheoutcomeofaclaim

isbasednotonlyonthecharacteristicsoftheclaimitself,butalsoontheoutcomesofother

claims,canimproveoutcomesgenerally.Thefirstarticleexaminestheconditionsunder

whichsamplingprocedurescanimproveaccuracyintheclassactioncontext,2whilethe

secondarticleexaminestheuseofcomparable-caseguidance(CCG),or“prior-award

information”—informationregardingawardsincomparablecasesasguidancefor

determiningdamageawards—toimproveaccuracyintheindividual-claimcontext.3

*HillelJ.Bavli&YangChen,ShrinkageEstimationintheAdjudicationofCivilDamageClaims,REV.OFL.&ECON.(forthcoming,2017).Inwritingthispaper,theauthorsbenefittedfromtheguidanceofDonaldB.RubinandJunS.Liu.

1HillelJ.Bavli,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyinClassActionLitigation.14LAW,PROBABILITY&RISK67(2015)[hereinafterAggregatingforAccuracy];HillelJ.Bavli,SamplingandReliabilityinClassActionLitigation,2016CARD.L.REV.DENOVO207[hereinafterSamplingandReliability].

2SeeAggregatingforAccuracy.

3HillelJ.Bavli,TheLogicofComparable-CaseGuidanceintheDeterminationofAwardsforPainandSufferingandPunitiveDamages,U.CIN.L.REV.(Forthcoming,2017)[hereinafterTheLogicofCCG].

115

Althoughsamplingandcomparable-caseguidancearequitedistinctinpractice,andarisein

differentcontexts,theunderlyingmechanismsbywhichtheyaffectaccuracyaresimilar.

Samplingproceduresinvolveadjudicatingaproportionofclaimsinaclassaction

(theclaimsinthe“samplegroup”)andextrapolatingdamageawardsfortheremaining

claims(theclaimsinthe“extrapolationgroup”).CCGmethodsinvolveincorporating

informationregardingawardsinpriorcomparablecasesintheadjudicationofadamages

awardinthesubjectcase.Samplingallowsforthesharingofinformationacrossclaimsina

class,whereasCCGallowsforthesharingofinformationacrossindividualclaims—but

bothmethodsaggregateanduseinformationregardingawardsincomparableclaimsto

influenceawardsofotherclaims.

Inthecurrentarticle,weexamineathird,butcloselyrelated—andinasense

unifying—formofclaimaggregationthatintegratessuchinfluenceexplicitly.Thisformof

claimaggregationisbasedonastatisticaldevicecalled“shrinkageestimation”(or

“shrinkage”),whichisusedinstatisticstoaggregateinformationandtherebyimprove

estimation.Specifically,itinvolvesadjustinganestimateofsomevaluetoaccountfor

informationderivedfromthepopulationofunitsfromwhichthatvalueisdrawn.4CCG,

whichusesinformationregardingcomparableclaimstoinfluencethesubjectclaim,canbe

understoodasaformofshrinkage.Similarly,samplingconstitutesaspecialcaseof

shrinkage,wherethepopulationofunitsistheclassofclaims,andtheinfluenceofthe

4SeegenerallyGeorgeCasella,AnIntroductiontoEmpiricalBayesDataAnalysis,39(2)J.OFTHEAM.STATIST.ASS’N83(1985);BradleyEfron&CarlMorris,DataAnalysisUsingStein’sEstimatorandItsGeneralizations,70J.OFTHEAM.STATIST.ASS’N311(1975);WillardJames&CharlesStein,EstimationwithQuadraticLoss,1PROCEEDINGSOFTHEFOURTHBERKELEYSYMPOSIUM361(1961).

116

informationderivedfromanindividualclaimentirelydeterminestheawardforeachclaim

inthesamplegroup,andtheinfluenceoftheinformationderivedfromsampledclaims

(ratherthananindividualclaim)entirelydeterminestheawardforeachclaiminthe

extrapolationgroup.

Ourobjectivesinthisarticlearetoexaminetheconditionsunderwhichshrinkage

canincreasetheaccuracyofdamageawardsintheclassactionandindividual-claim

contexts,andtoapplyshrinkagetogainadeeperunderstandingofthebenefitsand

limitationsoftheforegoingmethodswithrespecttoaccuracy.

WebegininSection2byreviewingthesamplingframeworkdevelopedin

AggregatingforAccuracy.InSection3,webuildonthisframeworktoexaminethebenefits

ofshrinkageintheclassactioncontext,andtoreexaminethebenefitsofsamplinginlight

ofshrinkageestimation.Weconsideralternativemethodologiesundervarious

assumptionsregardingcostandlegalconstraints.InSection4,weexamineconditions

underwhichshrinkagecanbeusedtoincreasetheaccuracyofdamageawardsinthe

individual-claimcontext.Inparticular,weconsidershrinkageinthecontextofproviding

jurorswithprior-awardinformationasguidanceindeterminingawardsforpainand

sufferingandpunitivedamages,andwederiveandillustratetheconditionsunderwhich

suchguidancecaninfactimproveaccuracy.InSection5,weconclude.

2. AFrameworkforExaminingSamplingandAccuracyinClassAction

Litigation

Inthissection,wesummarizetheframeworkdevelopedinAggregatingforAccuracy

andanumberofcentralresultsrelatedtotheuseofsamplingtoimproveaccuracyinclass

117

actionlitigation—firstinthecontextofahomogeneousclass,andtheninthecontextofa

heterogeneousclass.

2.1 SamplinginaClassofHomogeneousClaims

AggregatingforAccuracydevelopsaframeworkforexaminingtheeffectofsampling

onaccuracyinclassactionlitigation.Thearticleexaminesaprocedureinwhich1)a

numberofclaimsaresampledfromaclassofclaimsforindividualizedadjudication(and

receiptofindividualizeddamageawards),and2)themeanoftheawardsadjudicatedinthe

samplegroupisappliedastheawardforallremainingclaims,theclaimsinthe

extrapolationgroup.

Thearticle’sanalysisisintendedtorespondtoargumentsthatsuchprocedures

increaseefficiency(byallowingtheclaimantstoproceedasaclassratherthanas

individualclaimants),butonlyatthecostofreducingreliability.Itbuildsonassertionsput

forthbyProfessorsSaksandBlanckina1992StanfordLawReviewarticletoarguethat,

undercertainconditions,samplingandextrapolationmethodscanincreasereliabilityby

reducingerrorassociatedwithjudgmentvariability—uncertaintyresultingfromvariability

inthecompositionofajury,thepresentationofevidence,theselectionofajudge,etc.5

Toillustrate,considerhowreplicationmaybeusedtoreducejudgmentvariability:

[I]magine now a (costly) hypothetical procedure in which each and everyclaim[inagivenclass]werelitigatedtentimesindependently,andinwhichtheoutcomeassociatedwitheachclaimwerecomputedbytakingtheaverageofthetenverdictsassociatedwiththatclaim.Thatis,startbytakingthefirstclaimandlitigateitbeforetenindependentjuriestoobtaintenindependent

5SeeAggregatingforAccuracyat67-69;Michael.J.SaksandPeter.D.Blanck,JusticeImproved:TheUnrecognizedBenefitsofAggregationandSamplingintheTrialofMassTorts,44StanfordLawReview815,815-19(1992)[hereinafterJusticeImproved].

118

verdicts.Thenassigntheaverageofthetenverdictsastheoutcomeofthefirstcase.Byapplyingthisaggregatedoutcome,ratherthananysingleverdict,wemayreducetheerrorresultingfromjudgmentvariabilitytonearlynothing.6

AggregatingforAccuracyshowsthatsamplinginthecontextofaclassof

homogeneousclaims,throughitsimplicitorexplicituseofreplication,reducesjudgment

variabilityandtherebytoincreasesaccuracy.7Inparticular,thearticleconcludesthatgiven

aclassofNhomogeneousclaims,andlegalrestrictionsthatcanbedescribedby“reductive

sampling”—thatis,conditionsunderwhichthecourtwillnotreplaceanindividually

adjudicatedawardwithanawardextrapolatedfromotherclaims—accuracyismaximized

byrandomlyselectingasampleof𝑛∗ = 𝑁claimsforindividualizedadjudication,

assigningtheindividualizedawardsastheoutcomesoftheclaimsinthesamplegroup,

respectively,andthenapplyingthesamplemeanofthesample-groupawardsasthe

outcomeofallremaining(𝑁 − 𝑁)claimsintheclass.8

Tobemoreprecise:AssumewehaveaclassofNhomogeneousclaimsfromwhich

wesamplenclaims.Letusdefinea“correct”outcomeassociatedwithagivenclaimasthe

averageawardthatwouldemergefromrepeatedlitigationundervariouscircumstances,

suchasvariousjurycombinations,variouslawyers,judges,andpresentationsofevidence.9

The“correct”outcomecanbedefinedusingvariousmeasuresofcentraltendency,suchas

themeanormedian.Throughoutthepaper,weadoptthemeasureusedinAggregatingfor

6SeeAggregatingforAccuracyat77.

7Seeidat§4;JusticeImprovedat815-19.

8AggregatingforAccuracyat§4.

9AggregatingforAccuracyat§3;JusticeImprovedat833-34.

119

AccuracyandJusticeImproved—themean.Thus,letµbethecorrectawardineachoftheN

homogeneousclaims—themeanoftheawardsoverrepeatedlitigationofany(orall,since

theclaimsarehomogeneous)oftheNclaims.Andlet𝑋" bearandomvariabledefinedby

theadjudicatedawardintheithclaim,fori=1,2,3,...,n,wheretheXiareindependentand

identicallydistributed(i.i.d.)withmeanµandvarianceσ2,andthemeanoftheXi,fori=

1,2,3,...,n,isdefinedasXn,whichisdistributedwithmeanµandvariance,-

+.Thatis:

𝑋"~(𝜇, 𝜎))and𝑋+~(𝜇,,-

+).10Note,inAggregatingforAccuracy,itisassumedthattheXiare

distributednormally;butthisdistributionalassumptionisnotnecessaryfortheresults

derivedinthatpaperorinthecurrentpaper.Wethereforedropthenormalityassumption.

Thus,usingthesumofsquareresiduals

asthecriterionformeasuringerrorassociatedwithallNclaims,AggregatingforAccuracy

concludesthattotalerror,or“risk”—definedastheexpectationofRabove—isminimized,

andaccuracyismaximized,notbylitigatingeachclaimindividually(aprocedureoften

viewedbycourtsandscholarsastheideal,withrespecttoaccuracy),butratherby

samplingandindividuallyadjudicating

𝑁nop∗ = 𝑁

10AggregatingforAccuracyat§4.

120

claims,andapplyingthesamplemean𝑋+∗ astheoutcomeinallremaining(𝑁 − 𝑁)

claims,theclaimsinextrapolationgroup.11

Thus,inthecontextofahomogeneousclassofclaims,samplingmayimprove

accuracyaswellasefficiency.But,asexplainedinAggregatingforAccuracy,homogeneityis

notnecessaryforsamplingtoincreaseaccuracy.First,acourtmaystratifyaheterogeneous

classintomultiplerelativelyhomogeneoussubclasses.Forexample,theDistrictCourtfor

theEasternDistrictofTexas,inCiminov.Raymark,12usedsuchaprocedurewhenitdivided

aheterogeneousclassofasbestosclaimsintofivediseasecategories.Second,although

homogeneityishelpful,itisnotrequired—theerror-reducingbenefitsofsamplingapply

eventoaclassofheterogeneousclaims.

2.2 SamplinginaClassofHeterogeneousClaims

Homogeneityallowsthecourttorealizemaximalaccuracybysampling 𝑁claims.

Asaclassbecomesmoreheterogeneous,theutilityofthesamplingisreduced,sincethe

benefitsofreducingjudgmentvariabilitymustnowbebalancedwiththeerrorintroduced

byapplyingasinglepointestimate—thesamplemean—toaclassofheterogeneousclaims.

Thatis,thebenefitwithrespecttojudgmentvariabilitymustbebalancedwiththecost

withrespecttoclaimvariability.Thus,underconditionsofheterogeneousclaims,the

optimalsamplesizewillfallbetween 𝑁andN.13Simplystated,extrapolatingoutcomesof

11Id.

12751F.Supp.649(E.D.Tex.1990).

13AggregatingforAccuracyat§4.

121

unsampledclaimscanincreaseaccuracysolongastheheterogeneityoftheclaimsisnot

toolarge.

AggregatingforAccuracymodelsheterogeneousawardsasdrawsfromanormal

distributionwithmeansµiandvarianceσ2.Thatis,contrarytoahomogeneousclass,where

allclaimshavethesamecorrectawardµ,inaheterogeneousclass,eachclaimihasa

correctawardµi.Thus,Xi∼(µi,σ2),independent,whereµi∼(µ0,τ2),i.i.d.Note,hereagain

werelaxthenormalityassumptionusedinAggregatingforAccuracy.

Thus,againusingthesumofsquareresiduals,wehave

𝑅 = (𝑋" − 𝜇"))+

"23

+ 𝑋+ − 𝜇F)

8

F2+G3

asthecriterionformeasuringtheerrorassociatedwithallNclaims.Aggregatingfor

Accuracyminimizestheexpectationofthisexpression,andderivestheoptimalsamplesize

underconditionsofheterogeneousclaimstobe

.

Thus,whenτ2,theclaimvariability,isdominatedbyσ2,thejudgmentvariability,n∗falls

between 𝑁andN.Iftheclaimvariabilityisgreaterthanjudgmentvariability(τ2≥𝜎)),

thenn∗=N.Andwhentheclaimvariabilityiszero(τ2=0),then𝑛∗ = 𝑁,theresult

obtainedforahomogeneousclass.14

14Id.

122

3. ShrinkageintheClassActionContext

Thebestaggregationmethod,withrespecttoaccuracy,shouldbeconsideredin

lightofrelevantlegalandcostconstraints.Asindicatedearlier,theframeworkand

conclusionsdescribedaboveassumethatacourtmaynotreplaceanadjudicatedaward

withanextrapolatedaward(anassumptionreferredtoinAggregatingforAccuracyas

“reductivesampling”).AggregatingforAccuracyarguesthat,whilethereisclearprecedent

foraggregationandextrapolation,replacingindividuallyadjudicatedawardswith

extrapolatedawardsislikelyviolativeoftheConstitutionaswellasrulesandnormsofcivil

procedure.Intheabsenceofthisconstraint,however,otheraggregationmethodsmaybe

morebeneficialwithrespecttoaccuracy.Forexample,assumingnolegalorcost

constraints,acourtmayadjudicateallclaimsindividuallyandthenreplaceallindividual

awardswithextrapolatedawards,suchaswiththemeanoftheindividualawards.

Inthecurrentsection,werelaxthelegalconstraintspresumedinAggregatingfor

Accuracyinordertoconsidertheeffectofshrinkage—whichinvolvesthereplacementof

anadjudicatedawardwithonethatisinfluencedbyawardsincomparableclaims—on

accuracyintheclassactioncontext.Relaxingtheseconstraintsisusefulforatleasttwo

reasons.First,theremaybecontextsinwhichsuchproceduresarepermissible.For

example,partiesmayoptforsuchproceduresinsettlementormediationcontexts.Second,

examiningtheeffectsofshrinkageallowsamorecompleteunderstandingofaggregationin

lightofrelevantlegalandcostconstraints.

Thus,inthecurrentsection,webeginbyshowingthat,inthecontextofaclassof

claims,shrinkageachievesgreateraccuracythanclassicalcase-by-caseadjudication.Our

123

pointofcomparisoniscase-by-caseadjudication(asinAggregatingforAccuracy),rather

thanatypicalrepresentativeaction,becausetheformerisoftenviewedastheideal,with

respecttoreliability,andisusedastheprimaryalternativetoclasscertificationifputative

classrepresentativesareunabletoshowthatclasstreatmentisappropriate.Wethen

reexaminethesamplingresultsderivedinAggregatingforAccuracyusingshrinkage,and

showthatrelaxingthereductivesamplingconstraintandapplyingshrinkageleadsto

greateraccuracythaneventhesamplingprocedureexaminedinthatpaper.

3.1 ComparisontoIndividualizedAdjudication

Ourobjectiveinthissectionistoshowthat,givenaclassofclaims,replacingan

adjudicateddamagesawardwithanawardbasedonshrinkageincreasesaccuracy,in

expectation,foreachadjudicatedclaim(and,therefore,intheaggregate,overallsampled

claims).

Asabove,assumewehaveaclassofNclaimsandthateachclaimiisassociatedwith

acorrectawardµi,fori=1,2,3,...,N.Assumetheµi’sareequal(homogeneous)orthatthey

arisefromacommondistributionwithmeanµ0andvarianceτ2.Denotetheawardsofthen

sampledclaimsbyX1,...,Xn,andassumethattheyaredistributedaroundtheircorrect

awardsµi,respectively,withvarianceσ2.Thus,Xi∼(µi,σ2),independent,whereµi∼(µ0,τ2),

i.i.d.,andσ2andτ2areknownandrepresentjudgmentvariabilityandclaimvariability,

respectively.(SeeAggregatingforAccuracyforadiscussionregardingparameter

estimation).Asabove,wedonotrelyondistributionalassumptions.Ourobjectiveisto

imputeallofthemissingoutcomes ,whicharenotdirectlyobservable,using

124

estimatedvalues ,andtodosoinawaythatminimizeserror,representedbythe

(standard)riskfunction,

𝑅 = 𝐸 𝜇" − 𝜇" )8

"23

= 𝐸 𝜇" − 𝜇" )+

"23

+ 𝐸 𝜇" − 𝜇" )8

"2+G3

Wethusreplacetheclassicalestimator—anadjudicatedaward(Xi)—witha

shrinkageestimator(𝜇"),whichcombinestheadjudicatedawardwithadditional

informationobtainedfromtheotherclaimsintheclass,andtherebygeneratesamore

accurateaward.Wedefinetheshrinkageestimatoras:

.

Thisestimator(𝜇"U)forthecorrectoutcomeintheithclaimthusdiffersfromthe

classicalestimator,theadjudicatedaward(Xi),inthat𝜇"Uisaweightedaverageofthe

adjudicatedaward(Xi)andthemean(µ0)oftheµi,weightedbytheinverseofthe

variabilityofeach,respectively.Thus,Xiandµiareweightedbytheir“information”—the

inverseofthejudgmentvariability(σ2)andtheinverseoftheclaimvariability(τ2),

respectively.Intuitively,smallervariabilityimpliesgreaterinformation,andtherefore

heavierweight.15

Note,ifweassumethatthedistributionsofXiandµiareGaussian(or“Normal”)—

i.e.,ifweregardXi∼N(µi,σ2)asthelikelihoodandµi∼N(µ0,τ2)astheprior—weindeed

haveanexplicitstatisticaljustificationforthisestimator.ItistheBayesestimator,whichis

admissible.Itisalsothemaximumlikelihoodestimatorinthehierarchicalmodel.

15SeeCasella,supranote4.

125

Itfollowsthattheriskoftheshrinkageestimatoris

, (1)

whichissmallerthan𝑅"q :=E(Xi−µi)2=σ2,theriskassociatedwiththeclassicalestimatorXi

ofµi.

Therefore,foreachindividualclaim,assigningtheshrinkageestimatorasthe

damagesaward,onaverage,yieldsgreateraccuracy—thatis,lowerrisk—ascomparedto

theadjudicatedaward.Further,thisresultleadstotheconclusionthatapplyingthe

individualizedshrinkageaward𝜇"Utoeachclaimintheclassreducestotalrisk,sinceriskis

reducedwithrespecttoeachindividualclaim.

Now,ingeneral,wewillnotknowthevalueofµ0.Thus,substitutingitwiththe

unbiasedestimator𝜇r = 𝑋"/𝑛+"23 for𝜇rintheshrinkageestimator,wehavethe

“empirical”shrinkageestimator

, (2)

where𝜇r~(𝜇r,,-GJ-

+).NotethatintheGaussiancasewheretheXiareindependentandhave

distributionN(µi,σ2),andµihaspriordistributionN(µ0,τ2),thisestimatoriscalledthe

empiricalBayesestimator,16wherewereplacethehyperparameterµ0withitsmaximum

likelihoodestimate.ThisempiricalBayesestimator,whichconvergestotheBayes

16BradleyEfron&CarlMorris,Stein’sEstimationRuleandItsCompetitors—AnEmpiricalBayesApproach,68J.OFTHEAM.STATIST.ASS’N117(1973);HerbertRobbins,AnEmpiricalBayesApproachtoStatistics,1PROCEEDINGSOFTHETHIRDBERKELEYSYMPOSIUMMATHEMATICALSTATISTICSANDPROBABILITY157(1955).

126

estimatorasn→∞,isasymptoticallyadmissible.Therefore,statisticallyitisajustified

estimator.

Thus,letusconfirmthattheriskassociatedwiththisestimatorislessthantherisk

associatedwiththeclassicalestimator,anadjudicatedaward.Letting

𝐴 = ,u-

,u-GJu-,

wecanwritethis“empirical”shrinkageestimator𝜇" as𝜇" =AXi+(1−A)𝜇r.Theriskofthis

estimatoris

(3)

. (5)

ThisriskissomewhatlargerthanRiswithRise−Ris=n−1σ4/(σ2+τ2)>0.Intuitively,

sincewehavelessinformation—wedonotknowthetruevalueofµ0—wehavegreater

risk.However,asthesamplesizeincreases,thisriskconvergestothecaseinwhichwe

knowthetrueµ0.Thatis,Rise→Risasn→∞.

Importantly,however,althoughRise>Ris,Rise(theriskassociatedwiththeempirical

shrinkageestimator)issignificantlysmallerthanRic,theriskassociatedwiththeclassical

estimator(theadjudicatedaward).Thatis,Rise−Ric=(−1+n−1)σ4/(σ2+τ2)<0.Thus,under

thespecifiedcriterion,theempiricalshrinkageestimatorisalso(inadditiontothe

shrinkageestimator)abetterestimatorthananadjudicateaward.

127

Notethatifwedonothaveawayofestimatingclaimvariability(apossibility

discussedinAggregatingforAccuracy),wecanneverthelessrelyonStein’sestimator,17

whichdoesnotdependonclaimvariability:

,

where isanunbiasedestimatorfor(n−1)(σ2+τ2).IntheGaussiancase

(whereXiandµiareGaussian),(n−3)/Sisanunbiasedestimatorfor(σ2+τ2)−1.Therisk

associatedwithStein’sestimatorcanbederivedunderGaussianassumptions;18anditcan

beshownthatthisriskislessthantheriskassociatedwiththeclassicalestimatoraswell.

Indeed,Stein’sestimatorwillconvergetotheshrinkageestimatorasn→∞;andtherisk

associatedwithStein’sestimatorandtheshrinkageestimatorabovewillbeasymptotically

equal.

3.2 SamplingwithShrinkageEstimation

Inthissubsection,wereexaminetheaccuracybenefitsofsampling,butnowusing

theempiricalshrinkageestimator(𝜇"),ratherthananadjudicatedaward(Xi),fortheclaims

inthesamplegroup,and,onceagain,extrapolatingawardsfortheremainingclaims(the

claimsintheextrapolationgroup)usingtheestimatedglobalmean𝜇r.

Letusbeginbyderivingtheriskassociatedwiththeempiricalshrinkageestimator

inthecontextofthesamplingframeworkdiscussedabove.Thetotalriskforsampledand

non-sampledclaimsis:

17Efron&Morris,supranote4;James&Charles,supranote4.

18Efron&Morris,supranote4.

128

,

whereRise(Equation5)istheriskassociatedwithasinglesampledclaimiusingthe

empiricalshrinkageestimator(Equation2).

WecanthusseethattheriskRSisamonotonedecreasingfunctionofn:

. (6)

Thismeansthattheriskwillcontinuetodecreaseasweincreasethesamplesizen.

Ontheotherhand,ifweusetheclassicalestimator,anadjudicatedawardXi,to

estimateµi,weobtaintheriskfunction,

which,asderivedinAggregatingforAccuracyandreviewedabove,isminimizedat

. (7)

Thedivergenceoftheriskoftheempiricalshrinkageestimatorfromthatofthe

classicalestimatorisshowninFigureIV.1.FigureIV.1plotstheriskassociatedwithaclass

of1000claimsagainstthenumberofclaims,n,sampledforindividualadjudication.The

figureshowstheriskRCassociatedwiththeclassicalestimator(adjudicatedawards)and

theriskRSassociatedwiththeshrinkageestimatorforsamplesizesbetween10and200.

129

AsFigureIV.1illustrates,RCisminimizedpursuanttoequation7—wherehere,n∗is

approximately33(withσ/τ=5)—whereasRSismonotonicallydecreasinginn.AsFigure

IV.1suggests,ifwerelaxthereductivesamplingconstraintinAggregatingforAccuracy,and

insteadassumethepermissibilityofshrinkageestimation,thesamplesizethatminimizes

riskisfoundtobeN,thetotalnumberofclaimsintheclass.

(Note,ifclaimvariabilityisunknown,thecourtcanfirstsampleasmallnumberof

claimsforadjudicationanduse toestimate(n−1)(σ2+τ2).19Inthiscase,

.Wecanthenobtaintheasymptoticintervalforthisnewestimated

numberofclaimsusingthecentrallimittheoremandthedeltamethod.Wecantherefore

obtainarangefortheoptimalsamplesize,anddetermineourchoicebasedonsome

extraneouscriteria.)

19SeegenerallyAggregatingforAccuracy§§4-5.

130

NumberofSamples

FigureIV.1:Illustrationofrisk,plottedagainstthenumberofclaims,n,sampledfor

adjudication,comparingtherisk,RC,associatedwiththeclassicalestimator(adjudicated

awards),andrisk,RS,associatedwiththeshrinkageestimator,givenaclassof1000claims

andsamplesizesbetween10and200.

Importantly,theforegoingresultsshouldnotbeinterpretedassupportingan

argumentagainstsampling;thereliabilitybenefitsofsamplingareclearandsignificant

undertheconditions,includingthelegalconstraints,describedinAggregatingforAccuracy.

Rather,theseresultsdemonstratethebenefitsofshrinkage.Indeed,shrinkagedoesnot

detractfromthereliabilitybenefitsofsampling;rather,itsufficientlyenhancesthe

accuracyofindividualestimates—i.e.,resultingfromindividualizedadjudicationandthe

replacementofindividuallyadjudicatedawardswithshrinkageestimates—that,inasense,

itreducestheneedfor(i.e.,therelativebenefitsof)sampling.

50 100 150 200

Risk of 1000 claims

R C

R S

n * = 33

131

Furthermore,itissignificantthatincasesinwhichjudgmentvariabilitydominates

claimvariability(σ2>τ2),shrinkageestimationreducesriskatahighratewhenn<n∗and

atarelativelylowratewhenn>n∗(since,asnapproachesN,thesecondtermontheright

handsideofequation6approaches0,andthefirsttermdominates).Therefore,ifthe

reductivesamplingconstraintisrelaxed—and,inparticular,ifcourtsarewillingtoadjust

individualadjudicationstoincorporateaggregateinformation—then,tobalanceconcerns

foraccuracywithconcernsforlitigationcosts,acourtmayconstructasampling-basedtrial

planpursuanttotheresultsderivedinAggregatingforAccuracy,butaddtheadditional

stepofreplacingtheadjudicatedawardsofsampledclaimswithshrinkageestimates.That

is,thecourtmaysample

claimsforindividualizedadjudication;assignindividualizedshrinkageestimatestothen∗

sample-groupclaims(basedontheirrespectiveindividualizedawardsaswellasthemean

ofthesampledclaims);andapplythemeanoftheawardsinthesamplegroupasthe

outcomeofallnon-sampledclaims.Thisprocedureisidenticaltotheprocedurederivedin

AggregatingforAccuracy,withonedifference:here,thecourtwouldapplyindividualized

shrinkageestimates,ratherthanindividualizedclassicalawards,astheoutcomesofthe

claimsinthesamplegroup.

Inconcludingthissection,wenotethatwedonotintendtomakenormative

statementsregardingtheappropriateuseofshrinkageinlitigation.Forexample,itis

beyondthescopeofthispapertoaddresstheconstitutionalityofshrinkageorrelated

policyconcerns.Instead,wesimplyintendtodevelopamorecompleteunderstandingof

132

aggregation,withrespecttoreliability,andtoexamineanumberofkeyresultsregarding

thereliabilitybenefitsofshrinkage.

Givenourresultsabove,choosingtheaggregationapproachtomaximizeaccuracy

dependsontheapplicablelegalandcostconstraints.Forexample,ifthereisnoconcernfor

thereductivesamplingconstraintorcostconstraints,repeatedadjudicationsofeachclaim

inaclasscanleadtothegreatestdegreeofaccuracy.Ifthereisconcernforlitigationcosts

butnoconcernforthereductivesamplingconstraint,samplingaproportionofclaims

pursuanttoAggregatingforAccuracyandreplacingtheadjudicatedawardsinthesample

groupwithshrinkageestimates(andstillusingthemeanofthesamplegroupto

extrapolateawardsforallremainingclaims)maymaximizeaccuracy,giventheapplicable

costconstraints.Finally,ifthereisconcernforlegalconstraintsdescribedbyreductive

sampling(whetherornotthereisalsoconcernforlitigationcosts),thesamplingprocedure

(withoutshrinkage)derivedinAggregatingforAccuracyisthealternativethatmaximizes

accuracy.

Inthefollowingsection,weextendourdiscussiontocircumstancesoutsideofthe

classactioncontext.Inparticular,weconsiderthereliabilityimplicationsofshrinkagein

theindividualclaimcontext,whereinsteadofshrinkingtowardtheestimatedglobalmean

ofadefinedclassofclaims,weshrinktowardtheestimatedglobalmeanofapopulationof

awardsinpriorcomparablecases.

4. ShrinkageEstimationintheIndividual-ClaimContext

InSection3,wederivedthepotentialaccuracybenefitsofreplacinganadjudicated

awardwithashrinkageawardinthecontextofaclassof(homogeneousorheterogeneous)

133

claims.Weshowedthatshrinkagemaybebeneficialevenunderconditionsofhighclaim

variability.Theclassactioncontextprovidesaconvenientstartingpointfortheapplication

ofshrinkage,andaggregationproceduresgenerally,sincewearegivenapopulationof

claims(presumablywithrelativelylowclaimvariability)overwhichwearetoaggregate.

However,aheterogeneousclassofclaimsboundtogetherbysomecommonthreadshould

notbeunderstoodasfardifferent,forpurposesofshrinkage,fromapopulationofprior

claimsthataresimilarlyboundtogetherbysomecommonthread.Therefore,inthecurrent

section,weextendourdiscussionofshrinkagetotheindividual-claimcontext.

Forpurposesofthissection,therearetwomajorchallengestoapplyingshrinkagein

theindividual-claimcontext.Thefirstistheproblem,highlightedaboveinourdiscussionof

reductivesampling,thatgenerallyacourtcannotlegallyreplaceanadjudicateddamages

awardwithanawardextrapolatedformulaically.Thesecondistheproblemofidentifyinga

suitablesetofpriorcomparablecasestobeusedforshrinkage.

Itisbeyondthescopeofthecurrentpapertoexaminethelegalityofreplacingan

adjudicatedawardwithashrinkageaward.Rather,wecouchourdiscussionofshrinkage

hereinthecontextofarecentarticle,writtenbyoneoftheauthors,examiningprocedures

thataimtoreducethejudgmentvariabilityofcertaintypesof(particularlyunpredictable)

damageawardsbyprovidingafact-finder,priortoitsdeterminationoftheaward,with

informationregardingawardsinpriorcomparablecases.20Inparticular,TheLogicofCCG

usesprinciplesofshrinkagetoexplaintheaccuracybenefitsofcomparable-caseguidance,

20TheLogicofCCGat§1.

134

beyondtheempiricalresultthatsuchguidancereducesvariability,andtoaddressa

numberofmajorchallengestosuchmethods.Providingafact-finderwithprior-award

informationmayserveasaninnovativewayofusingshrinkagetoimprovecertaintypesof

damageawards—afterall,itenablesafact-findertoincorporatepriorawardinformation

justasashrinkageestimatorwould.Themajordifferenceisthatashrinkageestimator

incorporatessuchinformationformulaically,whereas,withthemethodsexaminedinThe

LogicofCCG,wemustrelyoncertainbehavioralassumptions.

Asdescribedinthatpaper,thereissubstantialevidencethatprovidingjurorswith

suchinformationiseffectiveinreducingjudgmentvariabilityandinfluencingadamages

awardgenerally,butwhetherajurorcanbesaidtoactasashrinkageestimator(e.g.,

allowingprior-awardinformationtoinfluencetheawardinproportiontotheinverse

variabilityofsuchinformation)iscurrentlybeingstudiedinaseriesofrandomized

experiments.

Inthecurrentsection,weaddressthesecondchallenge—theproblemofidentifying

asetofpriorcomparablecases.Effectively,weassumethatthefactfinderinTheLogicof

CCGactsasashrinkageestimator,andweexaminetherobustnessofcomparable-case

guidance,withrespecttoaccuracy,tovariationsinthesetofpriorcomparablecases

identified.Inotherwords,ourconcernisthis:assumingthefact-finderacts“rationally,”in

thesenseofoptimallyusingprior-awardinformationasashrinkageestimatorwould,what

choicesofpriorcases(fromwhichprior-awardinformationisobtained)willincrease

accuracyinexpectation?Forexample,how“wrong”canasetofpriorawardsbebefore

135

prior-awardinformationreducesaccuracy?Thesequestionsareessentialfordetermining

policysurroundingsuchmethods.

Althoughwedonotknow,atthispoint,whetherjuriesactaspredicted,applying

shrinkageexplicitlytothiscontextenablesanunderstandingofthepotentialbenefits,and

someofthepotentialrisks,associatedwiththeuseofcomparable-caseguidance.Atthe

veryleast,ouranalysisenablesanunderstandingofthepotentialbenefitsofcomparable-

caseguidance,andtherobustnessofthesebenefitsto“incorrect”setsofpriorawards.

4.1 Background:TheUseofComparable-CaseGuidancetoReduceAward

Variability

TheproblemaddressedinTheLogicofCCGistheunpredictability(i.e.,judgement

variability)ofawardsforpainandsufferingandpunitivedamages—twotypesofawards

forwhichthejuryreceivesverylittleguidanceindeterminingtheaward.TheSupreme

Courtandlowercourtshaverepeatedlyexpressedtheimportanceofreducingthe

variabilityofsuchawards.21

Thepaperhighlightstheproblemswithrelyingonexistingmethods,suchasadditur

andremittitur,toolsusedbycourtstoincreaseordecreasetheamountofanawardfound

tobeinadequateorexcessive.Althoughthesetoolscanbeuseful,andcanbeusedto

incorporateprior-awardinformationinvariousways,22andinconjunctionwithother

methods,alonetheyaddressextremeawardsonly,ratherthanvariabilitygenerally,and(in

21TheLogicofCCGat§2.See,e.g.,ExxonShippingCo.v.Baker,554U.S.471(2008).

22SeegenerallyJosephB.Kadane,CalculatingRemittiturs,8LAW,PROBABILITY&RISK125(2009).

136

practice)theyordinarilyaddressonlyexcessiveawards,ratherthaninadequateones.23

Additionally,widespreaduseofsuchmethodsarguablyreplacesthefact-finder’sdiscretion

withthatofthecourt,raisingconstitutionalandpolicyissues.Othermethods,suchascaps,

arbitrarilydrawcutoffpoints,leadingtosubstantialbiasandperverseoutcomes.24

Asmentionedabove,thereisempiricalevidencethatprovidingthefactfinderwith

informationregardingawardsinpriorcasesiseffectiveinreducingvariability.Butsuch

studiesdonotaddresstheeffectofprior-awardinformationonaccuracy—thatis,biasand

variability.

TheLogicofCCGestablishesaframeworkforexaminingthebenefitsandlimitations

ofprior-awardinformationintermsofaccuracy;anditaddressesanumberofmajor

challengestotheuseofprior-awardinformationtoreducevariability,includingthe

possibilityofusingawardinformationfroman“incorrect”setofcases.25Inthecurrent

section,inapplyingshrinkageestimationtotheindividualclaimcontext,weanalyze,and

deriveanumberofsignificantresultsregarding,thislatterchallengeinparticular.

4.2 IdentifyingPriorCases

Asapreliminarymatter,itisimportanttorealizethatthereisno“correct”or

“incorrect”setofpriorcases.AsdiscussedinTheLogicofCCG,theeffectofprior-award

informationonaccuracydependson1)thealignmentofmaterialfactsandissuesinthe

subjectcasetothoseinpriorcases;2)thesubstantivebreadthofthepriorcases;and3)the

23TheLogicofCCG§3.

24Id.

25SeeId.at§1.

137

numberofpriorcases—orthe“samplesize.”Forexample,thealignmentoffactsandissues

affectsthebiasintroducedbythepriorawards—wewouldlikefortheaveragecorrect

awardinthepriorcasestoalignwiththecorrectawardinthesubjectcase.Thebreadthof

thepriorawardsaffects,e.g.,theinfluenceoftheprior-awardinformationonthesubject

award;butasetofpriorcasesthatcontainsonlyidentical,oralmostidentical,material

factsandissuesmayresultinasamplesizeofoneortwo,orevenzero,priorawards.Thus,

inidentifyingasetofpriorcases,acourtmustbalanceitsinterestsinmaintaininga

reasonablesamplesize,areasonablebreadth,andcasesthatinvolvefactsandissuesthat

arerelativelyalignedwiththoseinthesubjectcase.26

ConsidertheexampleoftheSeventhCircuitcase,Jutzi-Johnsonv.UnitedStates,27

describedinTheLogicofCCG,involvinganawardforpainandsufferingarisingfrom

circumstancesinwhichajailinmatecommittedsuicidebyhanging,duetoafailureofthe

jailtosupervisehimappropriately.Acourtconsideringpriorawards(asJudgePosnerof

theSeventhCircuitdid)mightconsiderwhethertouseonlycasesinvolvinginmateswho

hungthemselves,peoplewhohungthemselvesfromthegeneralpopulation,peoplewho

committedsuicidefromthegeneralpopulation,peoplewhosufferedfromasphyxiation

(e.g.,drowning)fromthegeneralpopulation,etc.28Ifthecourtrestrictsitsconsiderationto

casesinvolvinginmateswhohungthemselves,itmayobtainonlyatinysamplesize;ifthe

26Id.at§§4-5.

27263F.3d753(7thCir.2001).

28Seeid.at760-61.

138

courtusesawiderbreadthofcases,thepriorawardswillhavelessinfluence,andhigher

riskofintroducingbias.29

Foramoredetaileddiscussionoftheseconsiderations,seeTheLogicofCCG.The

importantpoint,forpurposesofthecurrentanalysis,isthattherearetradeoffsbetween

breadth,alignment,andsamplesize—combinationsofwhichcorrespondtovariouslevels

ofbiasandvariance,andthereforeaccuracy.Thepurposeofthecurrentanalysisisto

examinetheseeffectstounderstandtheconditionsunderwhichprior-awardinformation

increasesaccuracy.

Thus,assumethatwehaveanindividualclaimthatreceivesawardY.Yiscentered

atthecorrectawardµywithjudgementvariabilityσy2(assumedtobeknown).Thecorrect

awardµy,iscenteredatλ0withvariance .Thatis,thecorrectawardcanbeunderstoodas

asingleawardfromadistributioncontainingapopulationofawardsfromcomparable

claims.Thecorrectawardsofthecomparableclaims,aswellasthatofthesubjectclaim,

aredistributedaroundaglobalmeanλ0withvariability .Thus,

.

Tobeclear,instatisticalterms,by“comparable”claimsorcases,wemeantosuggest

thattheirawardssomehowarisefromthesameglobalmean.

Now,ifweknowtheglobalmean(λ0)andvariability( )associatedwiththis

population,theshrinkageestimatoris

.

29TheLogicofCCGat§5.

139

SimilartoEquation1,weknowthattheriskofthisestimatoris( ,whichis

smallerthanσ2,theriskassociatedwiththeclassicalestimatorYtoestimateµy.Noticethat

thisestimatorrequiresustoknowthevaluesofλ0andη0.Amorerealisticscenarioisone

inwhichwedonotknowthesevalues,andinsteadneedtoestimatethembasedonprior

awards.AssumethattheidentifiedsetofpriorcasesinvolvesawardsX1,...,XNcenteredat

thecorrectawardsµ1,...,µNwithjudgementvariabilityσ2(assumedtobeknown).Thatis,

Wecanthenusethefollowingunbiasedestimatorstoestimateµ0and :

.

Wethereforeusethe“empirical”shrinkageestimator,

,

whichconvergestothe(non-empirical)shrinkageestimatorasN→∞.AsinSection3,

replacingλ0andη0withunbiasedestimatorsintroducessomeamountofuncertaintyinto

theestimatorandthusintroducessomerisk.However,asN→∞,theriskconvergestothat

ofthe(non-empirical)shrinkageestimator.Therefore,intermsofrisk,webenefitfroma

sufficientlylargeN,thenumberofpriorcases.Thus,ifthesetofpriorcasesistoosmall—in

thesensethattheriskresultingfromtheestimatedvaluesofλ0andη0dominatesthe

benefitsoftheempiricalshrinkageestimator—acourtcanincreasesamplesizeby

expandingthesubstantivebreadthofthepriorcases.

Inthefollowingsubsection,wediscussthebreadthandalignmentofpriorcases.

Although,forthereasonexplainedabove,samplesizeisanimportantconsiderationin

140

determiningbreadth,theaccuracybenefitsofshrinkagearefairlyrobusttolowsample

size.Specifically,becausetheshrinkageestimatorisinfluencedbysamplesizeonlythrough

theestimationofthemeanandvarianceofthedistributionofpriorawards,andbecause

thesequantitiescanbeestimatedreasonablywellwithasmallsamplesize,asampleof5to

15oftensuffices.Perhapsmostimportantly,asdiscussedindetailbelow,theaccuracy

benefitsofshrinkagearequiterobusttomisalignment;therefore,eveniftheestimation

turnsouttoberelativelypoor,shrinkagecangenerallybeexpectedtoimproveaccuracy.

Thisisespeciallytrue,inlightoftheremittiturdeviceandappellateprocess,bywhichthe

courtsmoderateextremeoutliers.Forthesereasons,althoughweremainconsciousof

samplesize,forsimplicity,itdoesnotplayacentralroleinourexamplesandillustrations

below.

4.3 BreadthandAlignmentofPriorCases

Weareinterestedinexaminingtwoconsiderations:breadthandalignment.To

illustrate,considerthecaseofJutzi-Johnson,discussedabove.JudgePosnerdisagreedwith

thepriorcasesidentifiedbyboththeplaintiffandthedefendant.Heexplainedthat“[t]he

plaintiffcitesthreecasesinwhichdamagesforpainandsufferingrangingfrom$600,000

to$1millionwereawarded,butineachonethepainandsufferingcontinuedforhours,not

minutes”;andthat“[t]hedefendantconfineditssearchforcomparablecasestoother

prisonsuicidecases,implyingthatprisonersexperiencepainandsufferingdifferentlyfrom

otherpersons,sothatitmakesmoresensetocompareJohnson’spainandsufferingtothat

ofaprisonerwhosufferedatoothachethantothatofafreepersonwhowasstrangled,and

concludingabsurdlythatanyawardforpainandsufferinginthiscasethatexceeded$5,000

141

wouldbeexcessive.”30JudgePosnerultimatelyconcludedthat“[t]hepartiesshouldhave

lookedatawardsinothercasesinvolvingasphyxiation,forexamplecasesofdrowning,

whicharenumerous.”31

Inthelanguageofthecurrentsection,JudgePosnerdisagreedwiththealignmentof

theplaintiff’scases,implyingthatawardscorrespondingtocasesinvolvinghours,rather

thanminutes,ofpainandsufferingwouldbeinappropriatelyhigh.Hedisagreedwiththe

breadth(andthealignment)ofthedefendant’scases,suggestingthatasetofcases

involvingthepainandsufferingofinmates,ratherthanthegeneralpopulation,istoo

narrow;andthatthedefendant’sfocusoninmatesledtoalignmentissuesthatresultedin

“absurd”conclusions.Additionally,JudgePosnerseemstosuggestthatthesetsofcases

identifiedbythepartiessufferedfromsmallsamplesizesaswell,indicatingthat

broadeningthepriorcasestoincludethoseinvolvingasphyxiationinthegeneral

populationwouldleadtonumerouscases.

Thus,consideracasethatreceivesanawardY,whichis“drawnfrom”adistribution

centeredatthecorrectawardµywithjudgementvariabilityσy2(known),whereµyis

centeredatλ0withvariance .Thatis,

.

AssumethatthecourtidentifiesasetofpriorcasesinvolvingawardsX1,...,XNwith

correctoutcomesµ1,...,µNcenteredatµ0withvarianceτ2.Thus,

30Jutzi-Johnson,263F.3dat760.

31Id.

142

Xi∼(µi,σ2),µi∼(µ0,τ2);i=1,...,N.

EssentiallythismeansXi∼(µ0,ψ2)for1≤i≤N,whereψ2=σ2+τ2.Now,consideran

estimatoroftheform

, (8)

whichweapproximatebyplugginginunbiasedestimators

,

fortheunknownparametersµ0andψ2=σ2+τ2,respectively,toobtain:

, (9)

Again,as .Thus,theriskofthisshrinkageestimator𝜇is

whichissmallerthanσy2when

.(10)

Letusconsiderthemeaningofthisconditionandthenexamineanumberof

numericalexamplestogainadeeperunderstandingoftheconditionsnecessaryfor

increasingaccuracy.Ontheright-handsideofEquation(10),ψ2=σ2+τ2isthetotal

variance(thesumofclaimvariabilityandjudgmentvariability)ofthepriorawards;andσy2

isthejudgmentvariabilityofthesubjectcase.Ontheleft-handsideofEquation(10),(µ0−

λ0)2isthesquareofthemisalignment(i.e.,thedifference)betweentheexpectedcorrect

143

awardinthesubjectcaseandthemeanofthepriorawards;and canbethoughtofasthe

claimvariabilityofthehypotheticalpopulationtowhichthesubjectcasebelongs.For

simplicity,wecanset =0andviewthecaseasitsownpopulation.Theconditionwould

besatisfied,forexample,if(1)thejudgmentvariabilityofthesubjectawardisgreaterthan

thehypotheticalclaimvariability(e.g.,where =0),and(2)thebreadthoftheprior

awardsisgreaterthantheirmisalignment.Ofcourse,theeffectsofonecanbeoffsetbythe

effectsoftheother.Forexample,theeffectsofextrememisalignmentcanbeoffsetbythe

effectsofextremejudgmentvariability.

Further,ingeneral,thegreaterthedispersionofthepriorawards,themore

“tolerance”thereisformisalignment.Ontheotherhand,higherprior-awardconcentration

requiresgreateralignment.Thus,itmaybebeneficialforthebreadthofthepriorawardsto

reflectthecourt’sconfidenceintheiralignmentwithrespecttothesubjectaward.

LetusconsideranexamplebasedondataobtainedinSaksetal.,32whichtestedthe

effects,withrespecttovariability,ofprovidingmockjurorswithcertaininformation

regardingpriorawards.Inonesetofcontrolconditionsinwhichmockjurorswere

providedwithafactpattern(basedonactualpersonalinjurycases)involvinga“high-

severityinjury,”abrokenback,themeanandstandarddeviationoftheawardamounts

determinedbyparticipantswasapproximately$3millionand$4million,respectively.33

Notethatthesevaluesarebasedonamountsdeterminedbymockjurorsratherthan

mockjuries.Also,however,“[b]ecausethedistributionfortherawdollarawardswas

32MichaelJ.Saksetal.,ReducingVariabilityinCivilJuryAwards,21LAWANDHUMANBEHAVIOR243(1997).

33Seeid.at249-53.

144

highlyvariableandpositivelyskewed,awardsgreaterthantwostandarddeviationsabove

themeanwererecodedtotheamountattwostandarddeviations.”34Saksetal.thereby

limitedthevariabilityofthedata.

BasedonthedatafromSaksetal.,weconstructFigureIV.2,whichassumesacorrect

award(µy)of$3millionandjudgmentvariability(σy)of$4million.Itisintuitivetoimagine

anapproximately“normal”distributionwithalmostallawardsfallingbetween0and$11

million(thatis,themean±2standarddeviations).Weassumealsothatµy=λ0=3million

andη0=0.Thatis,weviewthecaseasitsownpopulation,ratherthanadrawfroma

distributionofawards.Thus,thefigureillustratesrisk(definedhereinaserror)asa

functionofthemeanofthepriorawards(indicatingalignment).Wecansee,forexample,

thatifpriorawardsarecenteredat$4millionwithstandarddeviationequalto$2million,

wereduceriskby92%byusingtheshrinkageestimatorratherthantheclassical

estimator;andwecanexpecttheawardtofallwithintheinterval$74,000to$5.26million

(thatis,themean±2standarddeviations),ratherthan$0to$11million.Iftheprior

awardsarecenteredat$5millionwithvariability$3million,wereduceriskby76.8%,

relativetotheclassicalestimator;andwecanexpecttheawardtofallwithintheinterval$0

to$6.85million.Althoughnotshowninthefigure,itcanbecalculatedthat,forprioraward

meanandstandarddeviationequalto$8millionand$4million,respectively,wereduce

riskby36%,relativetotheclassicalestimator.Finally,ifthedistributionofpriorawards

hasameanandstandarddeviationequalto$3million(thecorrectaward)and$2million,

34Id.at249.

145

respectively,thevariability(intermsofstandarddeviation)oftheestimatoris$0.8million

ratherthan$4milliondollars;andriskisreducedby96%.

RiskofShrinkageEstimators(Black)&RiskofClassicalEstimator(Gray)

FigureIV.2:Comparisonoftheriskcorrespondingtotheshrinkageestimator(black

curves)andtheriskcorrespondingtotheclassicalestimator(grayhorizontalline)plotted

againstthemeanofthepriorawardswhenthecorrectawardis$3million(verticalblack

bar),andassumingη0=0(thesubjectcaseformsitsownpopulation)andjudgment

variabilityofthesubjectcase,σy,isequalto$4million.Theblackcurvescorrespondto

differentvaluesofprior-awardvariability,whichrangesfrom$0.2millionto$3million.

Notethat,althoughSaksetal.usedmockjurorsratherthanjuries,ourchoiceof

judgmentvariability—akeyfactorforwhetherprior-awardinformationcausesaccuracyto

increaseordecrease—islikelyconservative,sinceourchoice($4million)reflectsthe

methodologyinthatstudywherebyallawardamountsabovetwostandarddeviations

abovethemeanwerereducedtotheamountoftwostandarddeviations.Tobesafe,

Mean of prior awards (millions of dollars) 0 1 2 3 4 5 6 7

prior award variability = 0.2 prior award variability = 1 prior award variability = 2 prior award variability = 3

Risk of Classical Estimator

146

however,letusillustrateasecondexampleusinghalfthestandarddeviationusedaboveto

definejudgmentvariability.Thus,FigureIV.3assumesacorrectaward(µy)of$3million

andjudgmentvariability(σy)of$2million.Inthisexample,ifpriorawardsarecenteredat

$4millionwithastandarddeviationof$2million,wereduceriskby68.75%,relativetothe

classicalestimator.Ifthedistributionofpriorawardsiscenteredat$1millionwitha

standarddeviationof$400,000,adistributionthatisconcentratedaroundasignificantly

incorrectaward,weneverthelessreduceriskby7.5%,relativetotheclassicalestimator.

Finally,ifthedistributionofpriorawardshasameanandstandarddeviationequalto$3

million(thecorrectaward)and$2million,respectively,wereduceriskby75%,relativeto

theclassicalestimator.Furthermore,ifthevariabilityinthisfinalscenariowere$200,000,

theriskassociatedwiththeshrinkageestimatorisevensmaller—areductionof99.99%

(correspondingtoareductioninstandarddeviationfrom$4millionfortheclassical

estimatorto$20,000fortheshrinkageestimator).

147

FigureIV.3:Comparisonoftheriskcorrespondingtotheshrinkageestimator(black

curves)andtheriskcorrespondingtotheclassicalestimator(grayhorizontalline)plotted

againstthemeanofthepriorawardswhenthecorrectawardis$3million(verticalblack

bar),andassumingη0=0(thesubjectcaseformsitsownpopulation)andjudgment

variabilityofthesubjectcase,σy,isequalto$2million.Theblackcurvescorrespondto

differentvaluesofprior-awardvariability,whichrangesfrom$0.2millionto$3million.

Lastly,weconstructathirdfigureusingdatafromBovbjergetal.,35whichexamined

realawarddatabyseverityofinjurytoanalyzethevariabilityofawardsforpainand

suffering.Thedatapresentedherearearguablyveryconservativeaswell,since1)the

authorsaltogetherexcludedthe5%ofawardvaluesfarthestfromthemedian;2)thedata

includereportedincidentsofadditurandremittitur;and3)thedatareflectthevalueof

35RandallRBovbjergetal.,ValuingLifeandLimbinTort:SchedulingPainandSuffering,83NW.U.L.REV.,908(1988).

Risk of Shrinkage Estimators (black) & Risk of Classical Estimator (gray)

Mean of prior awards (millions of dollars) 0 1 2 3 4 5 6 7

prior award variability = 0.2 prior award variability = 1 prior award variability = 2 prior award variability = 3

Risk of Classical Estimator

148

dollarsin1987.36Thus,inFigureIV.4,weconsidertheexampleofseveritylevel7(outof9,

representingsevere,butnotmaximal,injuries),withmeanandstandarddeviationvalues

ofapproximately$2millionand$2million.Thegraphagaindisplaysdifferentcurves

correspondingtodifferentlevelsofprior-awardvariability.Italsoillustratestherisk

associatedwiththeclassicalestimatorasashadedhorizontalline(sincetheriskofthe

classicalestimatordoesnotdependonpriorawards)at$4millionsquared,whichisequal

tothestandarddeviationsquared.First,wecanseethatifpriorawardsarecenteredat$2

million(assumedtobethecorrectaward)withastandarddeviationof$500,000,we

reduceriskby99.65%,relativetotheclassicalestimator.Ifthepriorawardshaveamean

andstandarddeviationequalto$500,000and$500,000,wereduceriskby49.83%—a

milderreduction,duetotheintroductionofbias(butareductionnevertheless).Iftheprior

awardshaveameanandstandarddeviationequalto$500,000and$1.5million,

respectively,wereduceriskby64%—animprovementrelativetotheformerscenario,due

totheincreaseinbreadth,whichreducestheeffectofthebias.Ifthepriorawardshavea

meanandstandarddeviationequalto$4.2millionand$200,000,respectively,weincrease

riskby18.63%,sincewehaveatightlybounddistributioncenteredatasignificantly

incorrectawardvalue.

Ontheotherhand,ifthepriorawardshaveameanandstandarddeviationequalto

$4.2millionand$1.5million,wereduceriskby37.48%,since,now,theintroductionofbias

islessenedduetohighprior-awardbreadth,andthebeneficialeffectofpriorawardson

awardvariabilitydominates.

36Seeid.at919-24.

149

RiskofShrinkageEstimators(black)&RiskofClassicalEstimator(gray)

FigureIV.4:Comparisonoftheriskcorrespondingtotheshrinkageestimator(black

curves)andtheriskcorrespondingtotheclassicalestimator(grayhorizontalline)plotted

againstthemeanofthepriorawardswhenthecorrectawardis$2million(verticalblack

bar),andassumingη0=0(thesubjectcaseformsitsownpopulation)andjudgment

variabilityofthesubjectcase,σy,isequalto$2million.Theblackcurvescorrespondto

differentvaluesofprior-awardvariability,whichrangesfrom$0.2millionto$4million.

Usingthederivationsandexamplesabove,westatethefollowingconclusions:

1. Priorawardsthatarerelativelyalignedwiththecorrectawardcanleadtolargeaccuracy

benefits.

2. Higherbreadthofpriorawardsleadstoreducedinfluenceofpriorawards,andtherefore

reduced accuracy benefits—but improvements nevertheless. Considerations in

determiningappropriatebreadthalsoinclude1)confidenceinthealignmentoftheprior

awards,and2)obtaininganappropriatesamplesize.

Mean of prior awards (millions of dollars) 0 1 2 3 4 5

prior award variability = 0.2 prior award variability = 0.5 prior award variability = 1.5 prior award variability = 4

Risk of Classical Estimator

150

3. Priorawardsthataremisaligned—evensignificantlymisaligned—butareofhighbreadth

leadtoimprovements,butimprovementsthataresmallrelativetopriorawardsthatare

alignedandareoflowerbreadth.

4. Increasingonly thebreadthof theprior awards (without affectingalignment)will not

harmaccuracy,butwillonlyreducetheinfluenceofthepriorawards,andthereforetheir

benefitswithrespecttoaccuracy.

5. Priorawardsthataresignificantlymisalignedandareoflowbreadthcanleadtoharmful

effects on accuracy. However, harmful effects generally require prior awards that are

tightlyboundaroundasignificantlyincorrectaward.

Inshort,underrelativelymildconditions,theshrinkageestimatoroutperformsthe

classicalestimator,anadjudicatedaward.

5. Conclusion

Claimaggregationmethodsmayenableacourttoincreasetheaccuracyofdamage

awardsbyallowingforthesharingofinformationacrossclaims.Recentpapershave

arguedassuchinthecontextsof1)samplingaproportionofclaimsinaclassactionfor

purposesofextrapolatingawardsforunsampledclassclaims;and2)providingafact-

finderwithprior-awardinformationasguidanceindeterminingawardsforpainand

sufferingorpunitivedamages.

Ourgoalinthispaperwastoexaminecertainimplicationsofwhatcanbe

consideredathird,butcloselyrelated,formofclaimaggregationcalledshrinkage

estimation.Weanalyzedtheaccuracybenefitsofshrinkageinthecontextsofsamplingand

comparable-caseguidance;andweappliedittogainadeeperunderstandingofthebenefits

151

andlimitationsofclaimaggregationgenerally.Webeganouranalysisbyapplying

shrinkageintheclassactioncontext,andbyreexaminingtheresultsobtainedin

AggregatingforAccuracytoderivealternativemethods,withrespecttoaccuracy,under

variouslegalandcostconstraints.Wefoundthatshrinkageleadstogreateraccuracyas

comparedtoindividualadjudication,andalsoascomparedtothesamplingmethods

discussedinAggregatingforAccuracy.Butitalsorequiresrelaxingcertainlegalconstraints

thatareindeedapplicableinmanylegalcontexts.Wethereforediscussedalternatives

basedonvariouscombinationsofsuchconstraints.

Wethenextendedouranalysistotheindividualclaimcontext,applyingshrinkageto

gainadeeperunderstandingofthepotentialbenefitsandlimitationsofcomparable-case

guidance.Applyingcertainbehavioralassumptions,wederivedaformulathatexpresses

thepreciseconditions,intermsofthealignmentandbreadthofasetofpriorawards,

underwhichcomparable-caseguidanceleadstoanincreaseordecreaseinaccuracy.We

thenusedouranalysistodrawconclusionsregardingtherobustnessoftheaccuracy

benefitsofcomparable-caseguidancetovariationsinthesetofpriorawardsidentified,and

toillustrateitusinganumberoffiguresandexamples.

Shrinkageisanimportantconceptinstatistics.Althoughithas(unsurprisingly)

receivedlittleattentioninlaw,theideaofsharinginformationacrossclaimshasnumerous

applications.Ithasthepotentialtoplayasignificantroleinthelaw—atleastimplicitly,

suchaswiththeclaimaggregationmethodsdiscussed—toreduceawardvariabilityand

increasereliabilitygenerally.

152

V. GuidingJurorswithPrior-AwardInformation:ARandomized

Experiment*

(Coauthor:ReaganRose)

1. Introduction

Inciviljurytrials,juriesarefrequentlyaskedtodetermineawardsforpainand

sufferingandpunitivedamages.Buttheyreceivescantguidancetoassistthemintheir

assessments.Asaresult,thesedamageawardsarenotoriouslyvariable,underminingthe

law’sfairnessanddeterrenceobjectives,andcausingawiderangeofharmstoindustries

andindividuals.1Indeed,courtshavelongrecognizedandexpressedconcernforthis

“virtuallyunbridleddiscretion”2ofjuries,andtheneedtoaddressthis“standardless,

unguidedexerciseofdiscretionbythetrieroffact,reviewable...pursuanttonostandard

toguidethereviewingcourteither.”3

Arecentarticle,writtenbyoneoftheauthors,examinedthebenefitsofcomparable-

caseguidance(CCG),or“prior-awardinformation”—informationregardingawardsinprior

*HillelJ.Bavli&ReaganRose,GuidingJurorswithPrior-AwardInformation:ARandomizedExperiment.ThisresearchbenefitedimmenselyfromtheguidanceofProfessorDonaldRubin,andfromsupportprovidedbytheInstituteforQuantitativeSocialScienceandtheCenterforAmericanPoliticalStudies. 1SeePaynev.Jones,711F.3d85,94(2dCir.2013);seealsoExxonShippingCo.v.Baker,554U.S.471,499-503(2008).

2Geressyv.DigitalEquip.Corp.,980F.Supp.640,656(E.D.N.Y.1997)(quotingLeslieA.Rubin,ConfrontingaNewObstacletoReproductiveChoice:EncouragingtheDevelopmentofRU-486ThroughReformofProductsLiabilityLaw,18N.Y.U.REV.L.&SOC.CHANGE131,146(1990-91)).

3Jutzi-Johnsonv.UnitedStates,263F.3d753,759(7thCir.2001);seeHillelJ.Bavli,TheLogicofComparable-CaseGuidanceintheDeterminationofAwardsforPainandSufferingandPunitiveDamages,U.CIN.L.REV.at5-8(forthcoming,2017)[hereinafterTheLogicofCCG].

153

comparablecasesasguidancefordeterminingdamageawards—asanapproachto

addressingtheunpredictabilityofawardsforpainandsufferingandpunitivedamages.4

ThearticleconcludedthatCCGisgenerallyeffectiveinreducingunpredictabilityand

improvingtheaccuracyofawardsforpainandsufferingandpunitivedamages.5However,

theargumentforCCGmethods—thoseproposedinTheLogicofCCGandbyprevious

commentators—relyoncertainuntestedbehavioralassumptions.

Specifically,theprimaryobjectiontoCCGmethodsisthefollowing:“ifourproblem

istheunpredictabilityofawardscausedbyafact-finder’sinabilitytoassessobjectivelythe

appropriatevalueofawardsforpainandsufferingorpunitivedamages,howisitbeneficial

toprovideafact-finderwithinformationregardingdamagesawardedinpriorcasesthat

areseparateanddistinctfromthepresentcaseandthatpresumablysufferfromthesame

arbitrarinessthatwewishtoaddressinthepresentcase?”6TheLogicofCCGaddressedthis

objection,demonstratingthat,undercertainassumptionsregardingthebehaviorofatrier

offactinresponsetoprior-awardinformation,CCGcanimproveawardsforpainand

sufferingandpunitivedamagesunderarobustrangeofconditions.

Thus,theobjectiveofthecurrentstudyistoexaminetheeffectofCCGonawardsfor

painandsufferingandpunitivedamages.Inparticular,wepresentandinterprettheresults

ofafactorialexperimentdesignedtotesttheeffectsofCCGatdifferentlevelsofbias,

variability,andformofpresentation,aswellasinteractiveeffects,onthemagnitude,spread,

4SeegenerallyTheLogicofCCG.

5Seeid.at11-21.

6Id.at5.

154

andaccuracyofawardsforpainandsufferingandpunitivedamages.Weexaminethe

behavioroftriersoffactinresponsetoCCG,andwhetherCCGimprovesaccuracy—thatis,

whetheritsbeneficialeffectonthedispersionofawardsoutweighsanyerrorresulting

fromthedistortionofthemagnitudeofawards—underarobustsetofbias,variability,and

formconditions.

Insummary,wefindstrongevidencethatprior-awardinformationimproves

accuracy,andthatitsbeneficialeffectonthedispersionofawardsgenerallydominatesany

distortionary,orbiasing,effectonawards.Furthermore,wefindstrongevidencethattriers

offactrespondtoprior-awardinformationaspredictedinTheLogicofCCG,andinline

withthe“optimal”useofsuchinformation.

Section2providesabriefoverviewoftheproblem—thevariabilityofawardsfor

painandsufferingandpunitivedamages—anddescribesanumberofmethods,andCCG

methodsinparticular,proposedtoaddressit.Section3explainsthemethodologyofthe

presentresearch,andtheexperimentinparticular.Section4reportsandinterpretsour

results.AndSection5concludes.

2. AddressingtheVariabilityofAwardsforPainandSufferingandPunitive

DamageswithComparable-CaseGuidance

2.1 CurrentMethodsandProposalsforReducingVariability

Awardsforpainandsufferingandpunitivedamagesarenotoriouslyunpredictable.7

TheSupremeCourt,lowercourts,andnumerouscommentatorshaveexpressedconcern

7SeeRandallR.Bovbjergetal.,ValuingLifeandLimbinTort:Scheduling”PainandSuffering,”83NW.U.L.REV.908,919-24(1989)(“Tortlaw’straditionalmethodsofcomputingdamagesforpersonalinjuryanddeathareunderattack—andunderstandablyso.Legalreformershavelongarguedthatpresentlaw,whencombinedwithjurydiscretion,inflatesdamageawardsandcreatesproblematicoutcomevariability.Theopen-ended

155

forthevariabilityofsuchawardsongroundsoffairness,deterrence,andtheruinous

effectsofsuchawardsonbusinessesandindustries,andonsociety’strustinthecourts.8

Courtsandcommentatorshaveproposedandimplementedvariousmethodsfor

addressingtheunpredictabilityofawardsforpainandsufferingandpunitivedamages,but

nonehasprevailedasbothadequateandappropriate.

andunpredictablenatureoftortexposurehas,inturn,threatenedtheliabilityinsurancesystemthatfundsmosttortcompensation.”);ShariSeidmanDiamondetal.,JurorJudgmentsAboutLiabilityandDamages:SourcesofVariabilityandWaystoIncreaseConsistency,48DEPAULL.REV.301,317(1998)(examiningthe“considerablevariationinbothjurorandjuryawards”);OscarG.Chase,HelpingJurorsDeterminePainandSufferingAwards,23HOFSTRAL.REV.763,768-69(1995)(citingstudies);DavidW.Leebron,FinalMoments:DamagesForPainandSufferingPriortoDeath,64N.Y.U.L.REV.256,259(1989)(“Thedata...suggestthattortawardsforeventhisrelativelysimplearea[painandsufferingpriortodeath]varysignificantlyandthatneitherthespecificfactsofthecasenordifferingtheoreticalviewsofthefunctionsoftheawardscanexplainsuchvariation”);JoniHerschandW.KipViscusi,PunitiveDamages:HowJudgesandJuriesPerform,33J.LEG.STUD.1,1-3,4-10(2004)(examiningunpredictableawardsforpunitivedamages);seealsoTheLogicofCCGat1-5(citingcasesandliterature).ButseeTheodoreEisenbergetal.,VariabilityinPunitiveDamages:EmpiricallyAssessingExxonShippingCo.v.Baker,166JOURNALOFINSTITUTIONALANDTHEORETICALECONOMICS5(findingthatthedatadoes“notsupporttheunpredictabilityconcern”);Yun-chienChangetal.,PainandSufferingDamagesinPersonalInjuryCases:AnEmpiricalStudy,UNIV.OFCHICAGOCOASE-SANDORINST.FORLAW&ECON.ResearchPaperNo.749(April12,2016),http://papers.ssrn.com/abstract=2741180(“painandsufferingdamagesinTaiwanaretoalargeextentstatisticallyandlegallypredictable”).SeegenerallyNeilVidmar&MiryaHolman,TheFrequency,Predictability,andProportionalityofJuryAwardsofPunitiveDamagesinStateCourtsin2005:ANewAudit,43SUFFOLKU.L.REV.855(2010).

8See,e.g.,ExxonShippingCo.v.Baker,554U.S.471,499(2008)(“Therealproblem,itseems,isthestarkunpredictabilityofpunitiveawards”;“[t]hus,apenaltyshouldbereasonablypredictableinitsseverity,sothatevenJusticeHolmes’s‘badman’canlookaheadwithsomeabilitytoknowwhatthestakesareinchoosingonecourseofactionoranother.Andwhenthebadman’scounterpartsturnupfromtimetotime,thepenaltyschemetheyfaceoughttothreatenthemwithafairprobabilityofsufferinginlikedegreewhentheywreaklikedamage.”(citingOliverWendellHolmes,ThePathoftheLaw,10HARV.L.REV.457,459(1897)));BMWofNorthAmerica,Inc.v.Gore,517U.S.559,574(1996)(“Elementarynotionsoffairnessenshrinedinourconstitutionaljurisprudencedictatethatapersonreceivefairnoticenotonlyoftheconductthatwillsubjecthimtopunishment,butalsooftheseverityofthepenaltythataStatemayimpose”);Paynev.Jones,711F.3d85,94(2dCir.2013)(“Apartfromimpairingthefairness,predictabilityandproportionalityofthelegalsystem,judgmentsawardingunreasonableamountsasdamagesimposeharmful,burdensomecostsonsociety”);Geressyv.DigitalEquip.Corp.,980F.Supp.640,656(E.D.N.Y.1997)(commentingonthe“virtuallyunbridleddiscretion”ofjuriesindecidingawardsforpainandsuffering);Chase,supranote7,at768-69(“Variabilityisaproblemprimarilybecauseitunderminesthelegalsystem’sclaimthatlikecaseswillbetreatedalike;thepromiseofequaljusticeunderlawisanimportantjustificationforourlegalsystem.Variabilityisalsoclaimedtocreateinstrumentaldefects....”);Bovbjergetal.,supranote7,at908(“Determinationofawardsonanadhocandunpredictablebasis,especiallyfor‘non-economic’losses...tendstosubvertthecredibilityofawardsandhindertheefficientoperationofthetortlaw’sdeterrencefunction”);seealsoTheLogicofCCGat5-8(citingrelevantcasesandliterature).

156

First,courtscurrentlyusetheproceduresofadditurandremittitur—wherebya

courtthatfindsanawardtobeinadequateorexcessivemayorderanewtrialifthelitigant

harmedbytheproceduredoesnotagreetoanincrease(additur)orareduction

(remittitur)oftheaward.9Butthesedevicesaregenerallyinadequateastoolsfor

addressingvariability.Theyareappliedinconsistentlyandwithminimalstandards10;they

areusedtoaddressonlythemostextremeawardsratherthanvariabilitygenerally;and

regularuseofsuchmethods,andreplacementofjurydeterminationswiththoseofthe

court,wouldarguablyraisesignificantconstitutionalproblemsandbeinconsistentwith

fundamentalprinciplesoftortlaw.11Additionally,thesemethodsserveasa“band-aid”

ratherthanaddressingtheunderlyingproblem—thatjuriesreceiveinsufficientguidancein

assessingawardsforpainandsufferingandpunitivedamages.12

Second,numerousjurisdictionshaveimposeddamagecapstoaddressextreme

awards.Legislatureshaveenacteddamagecapsforcertaintypesofawards,suchas

9SeeDavidBaldusetal.,ImprovingJudicialOversightofJuryDamagesAssessments:AProposalfortheComparativeAdditur/RemittiturReviewofAwardsforNonpecuniaryHarmsandPunitiveDamages,80IOWAL.REV.1109,1118-20(1995).

10Jutzi-Johnsonv.UnitedStates,263F.3d753,759(7thCir.2001)(“[Mostcourts]treatthedeterminationofhowmuchdamagesforpainandsufferingtoawardasastandardless,unguidedexerciseofdiscretionbythetrieroffact,reviewableforabuseofdiscretionpursuanttonostandardtoguidethereviewingcourteither.”).

11SeeBaldus,supranote9;TheLogicofCCGat8-9.ButseeCassSunsteinetal.,PunitiveDamages:HowJuriesDecide248-52(2002)(arguingforalargerjudicialroleindeterminingpunitivedamages).Notethat,althoughourfocusisonmethodsthatmaintainthediscretionofthejury,asinTheLogicofCCG,ouranalysisregardingCCGmethodscaneasilybeextendedtoproceduresinwhichthecourthasamoresubstantialrole(onreview,orinthefirstinstance)indecidingawardsforpainandsufferingandpunitivedamages.

12Note,TheLogicofCCGproposestheuseofCCGinadditiontotheproceduresofadditurandremittitur,andnotinplaceofthem.

157

punitivedamages,ordamageawardsgenerally.13However,damagecapsaddressonly

extremecases,andonlyexcessiveawards.14Moreover,cappingawardsgenerallywithout

regardfortheindividualcircumstancesofacasegivesrisetofairnessandproportionality

concerns,andcanharmthedeterrenceobjectivesoftortlawanddisincentivizebeneficial

lawsuits.15Theymayraiseconstitutionalconcernsaswell.16

Third,anumberofcommentatorshaveproposedusingawardsincomparablecases

asguidanceforawarddeterminations.Thesemethodshavebeenproposedinvarious

forms.Somehavefocusedontheuseofcomparablecasesonreview,andtheimportanceof

creatingamoreprincipledapproachtoatrialcourt’sreviewofawardsforexcessiveness.17

However,althoughsuchmethodsimprovethestandardsunderlyingacourt’sreview,they

sufferfrommanyofthesameproblemsthatapplytotheproceduresofadditurand

remittiturgenerally.Forexample,theyaddressonlyextremeawards,andregular

replacementofthejury’sdiscretionwiththatofthecourtarguablygivesrisetoarangeof

constitutionalandsubstantivetortproblems.

Similarissuesarisefrommethodsthatinvolvebindingthetrieroffacttoa

particularawardorrangeofawards,orthatpredetermineascheduleofawardsinadvance

13SeeJosephSanders,WhyDoProposalsDesignedtoControlVariabilityinGeneralDamages(Generally)FallonDeafEars?(andWhyThisIsTooBad),55DEPAULL.REV.489,510(2006).

14TheLogicofCCGat9.

15SeeSanders,supranote13,at509-11;TheLogicofCCGat9.

16KathrynZeiler,TurningfromDamageCapstoInformationDisclosure:AnAlternativetoTortReform,5YALEJ.HEALTHPOL’Y,L.&ETHICS385,387(2005).

17See,e.g.,Baldus,supranote9.

158

ofacase.Thesemethodshavesimilarlybeencriticizedasreplacingthejury’sdiscretion

withthatofthecourtoralegislativebodyaltogetherremovedfromthecase.18

Somerecommendations,however,haveinvolved“comparabilityanalysis,”whereby

acourtidentifiescomparablecases,providesthetrieroffactwithinformationregarding

theawardsinthesecases,andinstructsthetrieroffacttoarriveatadamagesawardbased

ontheevidence,andusingthecomparable-caseinformationasguidance.19These

recommendationsarebasedonstudiesdemonstratingthattheyareeffectiveincontrolling

outlyingawardsandvariabilitygenerally(evenwhensuchinformationisprovidedasnon-

bindingguidance).20

2.2 Comparable-CaseGuidance

TheLogicofCCGexaminesaspecifictypeofcomparabilityanalysis.Itdefinesthe

termcomparable-caseguidance(CCG)as:

schedulingorcomparability-analysismethods that fulfill three fundamentalrequirements: 1) informationusedasguidancemustbederivedfromprior“comparable” cases (as opposed to, e.g., damage schedules predeterminedarbitrarily by a legislative body); 2) comparable-case informationmust beconsidered by the fact-finder in particular (as opposed to, e.g., a reviewingcourt);and3)comparable-caseinformationmustbeusedasguidanceonly(as

18TheLogicofCCGat10;seeDavidA.Logan,Juries,Judges,andthePoliticsofTortReform,83U.CIN.L.REV.903,942-43(2015)(“Suchanapproachwouldstreamlinelitigationandgreatlylimit,ifnoteliminate,theconcernswithvariabilityandfairnessthatthecurrentpracticerisksbytreatinglikecasesdifferently.However,thisapproachisfatallyflawedbecauseitevisceratesthevariouscontributionsthatjuriesmaketotheciviljusticesystem.Moreover,thisapproachisfundamentallyinconsistentwiththebasictortprinciplethateachvictimisentitledtoanawardtailoredtohisorhercircumstances,setbyalayjury.”(citingtheRestatement(Second)ofTorts§901cmt.a(1979))).

19TheLogicofCCGat3.

20SeeMichaelJ.Saksetal.,ReducingVariabilityinCivilJuryAwards,21LAWANDHUMANBEHAVIOR243,249-55(1997).

159

opposed to, e.g., imposing a range or amount that is binding on the fact-finder).21

Numerouscourtsandcommentatorshavecalledforsuchmethods.Consider,for

examplethecaseJutzi-Johnsonv.UnitedStates,whichinvolvedanappealfromanawardfor

painandsufferingresultingfromabenchtrial.22Inthatcase,JudgeRichardPosner

commentedonthe“acute”problemof“figuringouthowtovaluepainandsuffering.”23

AccordingtoJudgePosner,notwithstanding“varioussolutions,nonewhollysatisfactory,

[that]havebeensuggested,”“[m]ostcourtsdonotfollowanyoftheseapproaches.Instead

theytreatthedeterminationofhowmuchdamagesforpainandsufferingtoawardasa

standardless,unguidedexerciseofdiscretionbythetrieroffact,reviewableforabuseof

discretionpursuanttonostandardtoguidethereviewingcourteither.”24Heconcluded

that“[t]ominimizethearbitraryvarianceinawardsboundtoresultfromsuchathrow-up-

the-handsapproach,thetrieroffactshould,asisdoneroutinelyinEngland...beinformed

oftheamountsofpainandsufferingdamagesawardedinsimilarcases.”25Hecontinued:

“Andwhenthetrieroffactisajudge,heshouldberequiredaspartofhisRule52(a)

obligationtosetforthinhisopinionthedamagesawardsthatheconsideredcomparable,”

notingthatcourts“makesuchcomparisonsroutinelyinreviewingpainandsuffering

21TheLogicofCCGat4.

22263F.3d753(7thCir.2001)

23Id.at758.

24Id.at758-59.

25Id.(internalcitationsomitted).

160

awards,”andremarkingthat“[i]twouldbeawisepracticetofollowatthetriallevelas

well.”26

However,callsforsuchmethodshavegenerallyfailed.Indeed,therehavebeen

substantialobjectionstothesemethodsongroundsthattheirvalidityreliesontheability

ofacourttoreliablyidentifyanappropriatesetof“comparable”casesforguidancethat

wouldimprovetheawardathand.27ThefundamentalissueisnotwhetherCCGreduces

variability—itiswhetherCCGimprovesawards.

TheLogicofCCGthusaddressesafundamentalobjection:“howisitbeneficialto

provideafact-finderwithinformationregardingdamagesawardedinpriorcasesthatare

separateanddistinctfromthepresentcaseandthatpresumablysufferfromthesame

arbitrarinessthatwewishtoaddressinthepresentcase?”28Inessence,thepaperexplains

how“CCGmethodsreduceunpredictabilityandimproveawardsforpainandsufferingand

punitivedamagesbyallowingforthesharingofrelevantinformationacrosscases”;and

26Id.(internalcitationsomitted).SeealsoRoselleWissleretal,DecisionmakingAboutGeneralDamages:AComparisonofJurors,Judges,andLawyers,98MICH.L.REV.751(1999)(discussing“reformsconsistentwiththeavailabledata,”suggesting,“[a]notherpowerfulyetmodestreformwouldbetopooljuryawardsmadeforsimilarinjuries,andtopresentthesecasesandtheirawarddistributionstojuriesforguidanceinreachingtheirgeneraldamagesawardsandtojudgesforconductingtheiraddituir/remittiturreviews”);Chase,supranote7,at775,777-90(discussingrecommendationbytheABAActionCommissiontoImprovetheTortLiabilitySystemtoestablish“’tortawardcommissions’establishedtogatherandreportinformationthatwouldbeusefulin‘theframingofjuryinstructions,theexerciseofthepowerofadditurandremittitur,andtheprocessofsettlingcases,’”id.at775(quotingAmericanBarAssoc.,ReportoftheActionCommissiontoImprovetheTortLiabilitySystem10-15(1987)),andproposingmethodinvolvingchartsprovidingnonbindingguidance“toallowcomparisonwithroughlysimilarcasesinwhichplaintiffs’verdictswererecovered.”Id.at777-90);Sanders,supranote13,at496-507(describingproposalsandstudies);Logan,supranote18,at939-44(discussingproposals).

27SeeTheLogicofCCGat4-5.

28Id.at5.

161

howitdoessowithsuchconsistencythat(extraordinarycircumstancesaside)acourtneed

notbeconcernedwithwhetherCCGwillimproveaccuracyinaparticularcase.29

TheconclusionsinTheLogicofCCGarepremisedonthenotionthatfundamental

toagoodmethodofaddressingawardvariabilityistheabilityofthemethodtoreduce,

notonlyvariability,buterror:“reducingvariabilitydoesnotnecessarilymeanimproving

theaward.Forexample,itislikelyunwisetoencouragejurorstoanchortoanarbitrary

value,notwithstandingassociatedvariabilitybenefits.Infact,variabilitycouldbezeroif

thecourtweretodictateanawardwithoutregardtotheparticularsofthecase.”30Thus,

although“[c]ourtsandscholarsarecorrecttobeconcernedaboutthehighvariability

associatedwithawardsforpainandsufferingandpunitivedamages,”“goodpolicy

requiresstepstowardreducingsuchvariabilityonlyinsofarastheyimproveawards

generally.”31

TheLogicofCCG,therefore,buildsuponaframeworkforanalyzingthe“accuracy”

ofdamageawards.ItconcludesthatCCGcanimproveawardsforpainandsufferingand

punitivedamagesunderarobustsetofconditions—andspecifically,conditionsregarding

the“appropriateness”ofthepriorawardsidentified.Inotherwords,itconcludesthatthe

validityofCCGisnotverysensitiveto,andthereforedoesnotdependheavilyon,the

abilityofacourttoidentifyan“appropriate”setof“comparable”cases.

29Id.

30Id.

31Id.at11-12.

162

Butitsconclusionsregardingtherobustnessofsuchmethodsrelyoncertain

untestedbehavioralassumptions.Thecurrentpaper,therefore,aimstotestthe

propositionthatCCGnotonlyreducesthevariabilityofawards,butimprovesawards

underarobustsetofconditions.Ifsuchclaimsaresupportedbythedata,theymaydefuse

themajorobjectionstoCCGmethods,andmayweighheavilyinfavoroftheuseofthese

methodsinthecourtroom.

Inthefollowingsubsection,wedescribetheframeworkestablishedinTheLogicof

CCGforanalyzingtheaccuracybenefitsofCCG.

2.3 ReducingVariabilityandErrorwithCCG

TheLogicofCCGdefinesthe“correct”awardinacaseastheawardthatwouldresult

ifthecourthadperfectinformationregardingboththelawandthefactsofthecase.32It

explains,however,that“weneitherhaveperfectinformationnorknowthecorrectaward.

Instead,thecourtasksajurytoarriveatanaward,whichwillserveasanestimateofthe

correctoutcome.”33Itfurthercharacterizesthe“correct”awardas“themeanofthe

populationofpossibleawardsthatwouldemergefromadjudicatingthecaserepeatedly

undervariousconditions(e.g.,beforedifferentjudgesandjuries,bydifferentattorneys,

withdifferentpermutationsoffacts,etc.).”34Thus,“[a]singletrial…generatesasample

32Seeid.at12(citingHillelJ.Bavli,AggregatingforAccuracy:ACloserLookatSamplingandAccuracyinClassActionLitigation,14LAW,PROBABILITY&RISK67,74-78(2015)[hereinafterAggregatingforAccuracy]).Note,wecansimilarlydefineadistributionof“correct”awardsreflecting,e.g.,uncertaintyregardingthelaw.SeeHillelJ.Bavli,SamplingandReliabilityinClassActionLitigation,2016CARD.L.REV.DENOVO207,209n.16(2016).

33TheLogicofCCGat12.

34 Id. (citingAggregating forAccuracy at74-78(citingMichael J. Saks&PeterDavidBlanck, JusticeImproved:TheUnrecognizedBenefitsofAggregationandSamplingintheTrialofMassTorts,44STAN.L.REV.815,833-34(1992))).Thischaracterizationisintendedtocapturevariousinterpretationsofthe

163

fromthepopulationandanestimateofthecorrectaward.Calltheactualawardan

‘estimate,’andtheprocedurethatgeneratestheestimatean‘estimator.’Wecanthendefine

‘error’intermsofdistance,and(equivalently)‘accuracy’intermsofproximity,between

theestimateandthecorrectaward.”35The“reliability”ofalegalprocedureisthendefined

astheaccuracyoftheawardthatcanbeexpectedbyfollowingtheprocedure.36

Morespecifically,errorisdefinedbymeansquarederror(MSE),whichcanbe

separatedinto“bias”and“variance.”Biasrepresentssystematicerror,orthedifference

betweentheexpected,oraverage,awardandthe“correct”award.37Variance,ontheother

hand,isameasureofdispersion—thevariabilityoftheawardarounditsmean.Thus,

If theestimatorentailsahigh levelofvariance,but isunbiased, then itwillgenerateestimates[i.e.,awards]thatarehighlydispersedaroundthecorrectvalue. Inthiscase,wesaythattheestimatoris“unbiased,”butthatit lacks“precision.”Iftheestimatoris“precise”but“biased,”thenitgeneratesvaluesthat are tightly centered around the wrong value—an undesirablecircumstance.Ifanestimatoris“precise”and“unbiased,”thenitwillgenerateestimatesthatarecloseinproximitytothecorrectvalue,andwesaythatitis“accurate.”38

law,thefactsofthecase,norms,etc.Id.at12n.51.Thereareinfactmanyreasonabledefinitionsforthe “correct” award in a case. Even using the conceptualization above, other measures of centraltendency—suchas themedian—maybeused, dependingonourbeliefs regarding thebestway tocharacterize the “correct” award (e.g.,whetherwewant to capture information regardingextremevalues,etc.).Additionally,asdiscussedinfra§3,forpurposesofthecurrentstudy,itisnotnecessaryto define a “correct” award. We do so for purposes of clarity, and for consistency with previousliterature.

35Id.(citingAggregatingforAccuracyat74-78).

36Id.

37Id.at13.

38Id.

164

Therefore,error(andthusaccuracy)ismeasuredintermsofbiasandvariance.The

LogicofCCGmodelsawardsforpainandsufferingandpunitivedamagesformally,

examiningtwoformsofvariability—claimvariability,reflectingsubstantive(factual)

differencesfromclaimtoclaim,andjudgmentvariability,reflectingrandomvariation

associatedwiththeawardofagivenclaim.39Itexplainsthat“wewanttheestimateto

reflectthefirstformofvariability—thatmeaningfulfactualdifferences(e.g.,betweenthe

presentcaseandpriorcases)shouldresultindifferentawards.Ontheotherhand,wewant

tominimizethesecondformofvariability—therandomnessassociatedwiththe

determinationofdamagesinagivencase.”40ItdescribesthebenefitofCCGasfollows:

Unfortunately,sincewegenerallydonotknowthecorrectdamagesaward,itisimpossibletodistinguishtheformerformofvariabilityfromthelatter.Inotherwords,atradeoffarisesbetweenminimizingbias—thatourestimateddamagesawardshould,onaverage,beasclosetothecorrectdamagesawardas possible, reflecting the former form of variability—and minimizingvariance, the degree of randomness associated with the estimated award.Thus, in a sense, incorporating prior-award information as guidance indeterminingadamages awardallows the fact-finder (implicitly) to strike abalancebetweenbiasandvariancesoastominimizeerror.Byacceptingthepossibilityofintroducingsomebiasduetomaterialdifferencesbetweenthepresent case and prior cases, it is possible to gain far more, in terms ofreliability, due to the reduction in awardvariability that follows fromsuchguidance.41Butwhatbalanceshouldthetrieroffactstrikebetweenprior-awardinformation

andthefactsofthecaseathand?ConsidertwoextremescenariosdescribedinTheLogicof

CCG.First,assumethatthetrieroffactbasesitsawardentirelyonthepriorawards

39Id.at16(citingAggregatingforAccuracyat74-78).

40Id.at16-17.

41Id.at17.

165

providedtoit—say,forexample,thatitsimplyappliestheaverageofthepriorawardsas

itsawarddetermination.Inthiscase,therewouldbenoerrorcausedbyrandomvariation,

sincetheaverageofthepriorawardsisconstant,“buttotheextentthattheaverageofthe

priorawardsisdifferentfromthecorrectcurrentaward…therewillbesubstantialerror

frombias.”42Assume,ontheotherhand,thatthetrieroffactdecidesthatitwillnoteven

considertheprior-awardinformationaspartofitsawarddetermination—thatitwill

determineanawardbasedsolelyontheevidence.Inthiscase,assumingtheawardis

unbiasedinthefirstinstance,“therewillbenoerrorfrombiasbutsubstantialerrorfrom

variability.”43

Thus,“CCGimprovesthereliabilityofawardsforpainandsufferingandpunitive

damages—awardtypesthatsufferfromparticularlyhighdegreesofvariability—by

facilitatingabalancebetweenminimizingvariabilityandintroducingthepossibilityof

bias.”44

Mathematically,itispossibletoderivetheoptimallevelofinfluenceoftheprior-

awardinformation.Theidealinfluenceisbasedontherandomvariationoftheaward

determination(i.e.,judgmentvariability)andthevariabilityofthepriorawards(i.e.,

claimvariability).45“Higherjudgmentvariabilitysuggestsweakerinformationobtained

fromthepresentcaseandgreaterinfluenceofthepriorawards;higherclaimvariability

42Id.

43Id.

44Id.at18.

45Id.

166

suggestsweakerinformationobtainedfromthepriorcases(and,similarly,higher

potentialforbias)andgreaterinfluenceofthepresentcase.”46

Forexample,wherejudgmentvariabilityisextremelylow,andmultiple

adjudicationsyieldidenticaloutcomes,“thereislittletobegainedfromintroducing

prior-awardinformationandthepossibilityofbias.”47Wherejudgmentvariabilityis

high,however(asisoftenthecasewithawardsforpainandsufferingandpunitive

damages),thereismuchtobegainedfromprior-awardinformation.48Similarly,highly

variablepriorawardssuggest,e.g.,thepossibilityofsignificantdifferencesamongtheset

ofcases—relativetoeachotherandrelativetothesubjectcase—andlessguidancefor

theawarddetermination;whereaspriorawardswithlowvariabilityprovideclearer

information,and,inasense,strongerguidance.49

Thus,itistheoreticallypossibleto1)estimateclaimvariability,e.g.,bycomputing

thevarianceofthepriorawards,2)estimatejudgmentvariability,e.g.,bycomputingthe

varianceofawardsresultingfromrepeatedadjudications,and3)arriveatanaward(an

estimateofthe“correct”award)formulaicallyby,e.g.,computingaweightedaverageofthe

meanofthepriorawardsandthemeanoftherepeatedadjudications,weightedbythe

46Id.

47Id.

48Id.

49Id.

167

inverseofthevarianceofeach,respectively.50Statisticiansrefertothistypeofestimatoras

a“shrinkage”estimator.51

Buttwoproblemsarise.First,itisdifficultandcostlytocomputejudgment

variabilityandgenerallyimpossibletoknowwhetherasetofpriorawardsis“aligned”

with—orhasthesamecenteras—the“correct”awardinthesubjectcase.Therefore,a

courtwouldbeunabletodeterminetheappropriateinfluenceofthepriorawardsor

whetherCCGwouldbebeneficialornot(orevenharmful)fortheawardathandinthefirst

instance.Second,thebenefitsofCCGaregroundedinassumptionsthatatrieroffactwould

actaspredicted,andthatdoingsowouldindeedimproveaccuracy.Buthowdoweknow

thatsuchassumptionsarejustified?

TheLogicofCCGaddressesthefirstproblembyexplainingthatitisunnecessary

tocomputejudgmentvariabilityorwhetherCCGisbeneficialinaparticularcase.That

paperarguesthat,inlightofthevariabilityofawardsforpainandsufferingandpunitive

damages,andtherobustnessoftheaccuracybenefitsofCCGunderawiderangeof

conditions,includingthoseinvolvingpriorawardsthatarenot“aligned”withthe

“correct”award,courtsshouldtrustthat,absentextraordinarycircumstances,a

reasonableprocedureforidentifyingpriorawardswillresultinCCGhighlylikelyto

improveaccuracy.“Statistically(usingfundamentalsof‘shrinkageestimation’),

50SeeHillelJ.Bavli&YangChen,ShrinkageEstimationintheAdjudicationofCivilDamageClaims,REV.OFL.&ECON.(forthcoming,2017),foranexaminationofaformulaicapplicationofthisapproach.

51SeeTheLogicofCCGat15-16,18-19n.65.SeealsoBradleyEfron&CarlMorris,DataAnalysisUsingStein’sEstimatorandItsGeneralizations,70J.OFTHEAM.STATISTICALASS’N311,312-14(1975);BradleyEfron&CarlMorris,Stein’sParadoxinStatistics,236SCIENTIFICAMERICAN119(1977).

168

incorporatingprior-awardinformationwillgenerallyimprovereliability,aslongasthe

priorawardsarenottightlybound(i.e.,withlowvariability)aroundameanthatisfar

fromthecorrectawardinthepresentcase—inpractice,aslongasthepriorawardsare

nottightlyclusteredaroundanawardthatreflectsfactsmateriallydissimilartothe

presentcase.”52Thepaperconcludesthat“[i]nshort,wecanexpectthatareasonable

methodforidentifyingprior‘comparable’caseswillresultinprior-awardinformation

thatislikelytoimprovereliability.”53

Thesecondproblem,regardingwhetheratrieroffactwouldactaspredicted

inresponsetoCCG,andwhetherdoingsowouldinfactimproveaccuracyina

robustrangeofcircumstances—notwithstandingtheconcernsexpressedabove—is

thesubjectofourexaminationinthecurrentstudy.

Specifically,ourprimaryobjectivesaretoexamine1)whetherprior-award

informationimprovesaccuracyunderarobustsetofconditionsrelatedtothebias,

variability,andformofpriorawards,and2)whethertriersoffactrespondtoprior-

awardinformationaspredicted,astojustifytheassumptionsandconclusionsinThe

LogicofCCG.

Therehavebeenanumberofstudiesexaminingtheunpredictabilityof

awarddeterminationsandmethodsofaddressingit.54Thesurveyexperimentby

52TheLogicofCCGat20-21(internalcitationsomitted).

53Id.at21.

54See,e.g.,Saksetal.,supranote20;Diamondetal.,supranote8;Bovbjergetal,supranote7;Baldusetal.,supranote9;CatherineM.Sharkey,UnintendedConsequencesofMedicalMalpracticeDamagesCaps,80N.Y.U.L.REV.391(2005);Leebron,supranote7.SeealsoSanders,supranote13,at496-507(summarizingproposalsandstudies);Chase,supranote7.

169

ProfessorSaksetal.ismostcloselyrelatedtothecurrentstudy.55Thatstudy

investigatedtheeffectsofprior-awardinformation,invariousforms,ontheamount

awardedforpainandsufferinginpersonalinjurycases—andspecifically,with

respecttoawardvariabilityand“distortions”intheamountsawarded.56Prior-

awardformstestedintheirstudyincludean“interval,”an“average,”an“average-

plus-interval,”“examples,”anda“cap”condition.Thestudyinvolvedthreedifferent

“severity”levels,whichdeterminedtheseverityoftheinjuryatissueinafact-

patternprovidedtoparticipants.Each“guidancecondition”waspredetermined

basedontheresultsofapilotstudy.Forexample,theauthorsusedthemedianof

thepilotstudytodefinethe“average”condition,the10thand90thpercentilesto

definethe“interval”condition,the10th,50th,and90thpercentilestodefinethe

“average-plus-intervals”condition,andthe10th,35th,65th,and90thpercentilesto

definethe“examples”condition.57

Insummary,theauthorsfoundthatthe“interval,”“average-plus-interval,”

and“examples”conditionscausedareductioninvariability,whiledistortingaward

amountsminimallyornotatall.58Theyalsofoundthatthe“cap”condition

performedpoorlyandsometimesincreasedthevariabilityandsizeofawards.59

55SeeSaksetal.,supranote20,at246.

56Seeid.at246-47.

57Seeid.at248.

58Seeid.at246.

59Seeid.at253.

170

AlthoughthesubjectofthestudybySaksetal.,anditsuseofpilotdatatoestablish

guidanceconditions,aresimilartothoseofthepresentstudy,thedetailsofthedesignand

analysisofthepresentstudydiffersubstantiallyfromthoseofthestudybySaksetal.and

otherstudies.Mostfundamentally,ourresearchdiffersfromallpreviousstudiesinits

focusonaccuracy,aswellasspreadandmagnitude,underarobustsetofbias,variability,

andformconditions.Thepresentstudyisalsouniqueinitsfocusonthebehavioralaspects

ofthemodelpresentedinTheLogicofCCG,including,forexample,testingtheeffectof

prior-awardvariabilityonamountsawarded.

3. Methodology

Ourobjectiveistotesttheeffectsofprior-awardinformationatdifferentlevelsof

bias,variability,andform,andtheirinteractions,onthemagnitude,spread,and,accuracyof

awardsforpainandsufferingandpunitivedamages.Wetesttheseeffectsexperimentally

(i.e.,byimposinganintervention)usingafactorialdesignandthepotentialoutcomes

framework.60Thismeansthateachoftheparticipantsisexposedtoa“treatment

combination”(or“treatmentcondition”)—intheformofasurvey—arisingfromasetof

“factors,”eachofwhichcanbesettoacertainvalue,or“level,”61andthatwemakecausal

conclusionsbasedoninferencesaboutwhatanoutcomewouldbeunderexposureto

60SeeTirthankarDasgupta,NateshS.Pillai,&DonaldB.Rubin,CausalInferencefrom2KFactorialDesignsbyUsingPotentialOutcomes,77(4)J.OFTHEROYALSTAT.SOC.SERIESB(STAT.METHODOLOGY)727,727(2015)[hereinafter2KFactorialDesigns];DonaldB.Rubin,EstimatingCausalEffectsofTreatmentsinRandomizedandNonrandomizedStudies,66(5)J.OFED.PSYCH.688(1974).Weusethenotationandgeneralframeworkestablishedin2KFactorialDesignsbutextendthenotationtoaccommodatetheadditionallevelspresentinthisexperiment.

61Seegenerally2KFactorialDesigns.

171

alternativetreatmentcombinations.62Inthecurrentsection,wedescribeourmethodology,

includingthedetailsofourexperimentaldesign(“theDesign”)andouranalysis.

3.1 SurveyAdministration

WeusedAmazon’sMechanicalTurktorecruitandadministersurveysto5500

participantssimilartoindividualsfoundonU.S.juries(U.S.citizenswhoare18orolder

andEnglishspeaking).63Participantsenrolledintheexperimentsequentiallyandwere

randomlyassignedtoreceiveoneof22surveys,eachreflectingaspecifictreatment

combination,whichdeterminedwhatprior-awardinformation,ifany,theparticipant

receivedasguidanceinrespondingtothesurvey.64Ineachsurvey,participantswere

presentedwithafactpattern,instructions,andcertainguidancefordeterminingadamages

award,basedontheassignedtreatmentcombination.Theywerethenaskedtodeterminea

62SeegenerallyRubin,supranote60,at689-90;C.F.JeffWu&MichaelS.Hamada,Experiments:Planning,Analysis,andOptimization(2ded.2009).

63MechanicalTurkrequiresthatallparticipantsbe18orolderandEnglishspeaking.TheU.S.citizenshiprequirementwasdifficulttoenforce;butweattemptedtoenforceditbyrequiringthatallparticipantshaveanIPaddresslocatedwithintheU.S.Additionally,weprovidedinformationregardingeligibilityrequirementstoallpotentialparticipantsandaskedthattheyparticipateinthesurveyonlyiftheymetthestatedrequirements.Notethatweignoredsomequalificationsforjuryservice,suchastherequirementthattheparticipantnotbeaconvictedfelon.

64Toensureroughlyequalsamplesizesacrosstreatmentgroups,a“dynamicrandomizationscheme”wasused.Thisschemepreventsanysingletreatmentconditionfromhavingmorethantwomoreparticipantsthananyotherconditionatanystageoftherandomization.Forexample,ifthefirsttwoparticipantstoenrollinthestudywererandomlyassignedtocondition1,thenextparticipantwhoenrolledinthestudywouldberandomlyassignedtooneoftheremainingtwenty-oneconditions.Thisschemecanbeviewedasaformof“blockrandomization,”whichmaintainsthepropertiesofclassicalrandomization,assumingtheparticipantsenrollrandomly(inthesensethatoneindividual’senrollmentdoesnotaffectanother’ssubsequentenrollmentotherthanthroughtherestrictiononthenumberofunitsineachtreatmentcondition).

172

damagesaward,provideanoptionalexplanation,andprovidecertaindemographic

information.65Theprimaryoutcomeinthisstudyistheawardamount.66

3.2 TreatmentCombinations

ForpurposesoftheDesign,theexperimentcanbeunderstoodastwoseparate

studies:acontrolstudywherenopriorawardinformationisavailableandatreatment

studywherepriorawardinformationisprovided.67Thecontrolstudyhasone

experimentalfactor:scenario(2levels).Thetreatmentstudyhasfourexperimental

factors:scenario(2levels),form(3levels),bias(2levels),andvariability(2levels).

Scenariocanbesettooneoftwolevels:painandsufferingorpunitivedamages.

Formcanbesettooneofthreelevels:average,list,orrange.Biascanbesettooneoftwo

levels:unbiasedorbiased.AndVariabilitycanbesettooneoftwolevels:lowvariabilityor

highvariability.

Thus,eachtreatmentcombinationischaracterizedbytheleveltowhicheachfactor

isset;andthetreatmentcombinationtowhichaparticipantwasrandomizeddetermined

theparticularsurveyheorshereceived.Forexample,aparticipantrandomizedtothe

treatmentcombinationthatinvolves[scenario=painandsuffering,form=list,bias=

unbiased,andvariability=highvariability]receivedasurveythataskedtheparticipantto

65Participantsreceived$.20fortheirparticipationinthesurvey.

66Theoptionalexplanationfortheawardamountchosenwillbeanalyzedseparatelyinasecondstudy.Thisoutcomeisthereforenotaddressedinthisstudy,excepttoconfirmourexpectationthatsurveyparticipantsprovidedthoughtfulresponses,ratherthansimplyfillinginaresponsewithoutanythought.

67Thiscategorizationisconvenientforpurposesofconsideringtheexperimentalfactors.Ontheotherhand,asdiscussedfurtherbelow,ourresultsareanalyzed,andmostconvenientlyunderstood,withineachlevelofscenario.

173

determineanawardforpainandsuffering,andprovidedunbiased,highvariabilityprior-

awardinformationintheformofalist.

Bydesign,certainlevelsofcertainfactorsareincompatiblewithcertainlevelsof

otherfactors.Inparticular,inthetreatmentstudythevariabilityfactorisnotapplicable

whenprior-awardinformationispresentedintheaverageform.Also,aspreviouslystated,

inthecontrolstudyonlythescenariofactorisapplicable.

Thus,thefactorsdefinetwotreatmentconditionsinthecontrolstudyand2x3x2x

2=24treatmentconditionsinthetreatmentstudy,minusfourinapplicableconditionsthat

arisefrom“crossing”thevariabilityfactorwiththeaverageleveloftheformfactor,fora

totalofJ=2+24–4=22treatmentconditions.Hereafter,werefertotheconditionswithin

thecontrolstudyasthe“controlconditions”andtheconditionswithinthetreatmentstudy

asthe“activetreatmentconditions.”Allconditionsareexplainedinmoredetailbelow.

3.3 TheSurveys

Eachsurveyforthecontrolconditionscontainsathree-paragraphstem.Thefirst

paragraphcontainsgeneralinstructions:

Thefollowingsurveyshouldtake5-10minutes.Pleasereviewtheinformationbelow, and provide your response and the demographic informationrequested.Pleasedonotconsultanyoutsidesources.Thedatawecollectwillbeusedforresearchpurposes.Wewillnotattempttoidentifyyouorconnectyourresponsestoyouridentity.Ifyouridentitybecomesknowntous,itwillremainprivate.”

Thesecondparagraphcontainsashortfactpatternthatbrieflydescribesatortcommitted

byacorporationandtheconsequencesofthetortforitsvictim.Thethirdparagraph

containsthejudge’sinstructionsforarrivingatanaward.

Asdiscussedabove,thecontrolstudyinvolvesonlyasinglefactor,scenario,which

174

hastwolevels,punitivedamagesandpainandsuffering.Thelevelofscenariodetermines

thefactpatternandthejudge’sinstructionsinthesecondandthirdparagraphsofthestem.

Inparticular,thesecondandthirdparagraphsofthepunitivedamagesscenariostate:

Youareajurorinatrialinwhichacarmanufacturerconcealeditsknowledgeofadefectinitscar’sairbagsystem.Asaresultofthedefect,theairbagswouldfail to deploy in a small proportion of frontal collisions. The lawsuit wasbroughtbyadriver,Andrew,whosufferedseverebraininjuryfromafrontalcollision caused by ice on the road. He now liveswith headaches, blurredvision,speechimpairment,andmemoryloss.Attrial,itwasestablishedthat,asaresultofthedefect,theairbagsfailedtodeploy.Itwasalsoestablishedthat,hadtheairbagsdeployedproperly,Andrew’sinjurieswouldhavebeenavoided.Thejudgehasaskedyoutodeterminea“punitivedamages”award.Heinformsyou that, through a separate proceeding, Andrew has already beencompensatedforhisinjuries,includinghismedicalexpensesandhispainandsuffering. The judge instructs you that your role now is to determine a“punitivedamages”award.Heexplainsthat“punitivedamagesaredamagesawarded not to compensate the plaintiff for any injury but to punish thedefendantforoutrageousconductandtodeterthedefendantandothersfromsimilarconductinthefuture.Youarenotrequiredtoawardpunitivedamages,andyoumayawardsuchdamagesonlyifyoufindthatthedefendant’sconductwasinfactoutrageous.”Thejudgeemphasizesthat“thereisnoexactstandardfordeterminingpunitivedamages.Youshoulddecideonanamountthatyoufind necessary for achieving the objectives described above. You shouldconsiderthedegreeofreprehensibilityofthedefendant’smisconductandtheactualorpotentialharmsufferedbytheplaintiff.”

Thesecondandthirdparagraphsofthepainandsufferingscenariostate:

You are a juror in a trial inwhich a company intentionally disposed of itsindustrialwastebyregularlydumpingitintoalocalriverratherthanhavingtheexpenseofdisposing itproperly. The lawsuitwasbroughtbyEmma,a29-year-old woman whose drinking water was affected by the improperdisposalandwhodevelopedararecancerasaresult.Threeyearsbeforeherdiagnosis,Emmamarriedhercollegeboyfriend. Sheandherhusbandnowhaveatwo-year-olddaughter.Emmahasundergonemultiplesurgeriesandmonths of chemotherapy and radiation therapy, but doctors have recentlyinformedher that thecancerhasspreadand thather likelihoodofsurvivalbeyond sixmonths is very low. Since her diagnosis one year ago, she hassufferedfromregularpain,nausea,fatigue,anddisfigurement,andherorgans

175

haverecentlybeguntofail.ThejudgehasaskedyoutodetermineasuitabledamagesawardforEmma’spainandsuffering(pastandfuture)andherlossofcapacityforenjoymentoflife.Heinformsyouthat,throughaseparateproceeding,Emmahasalreadybeencompensated forhermonetarycosts, suchaspastand futuremedicalexpenses.ThejudgeinstructsyouthatyourrolenowistodetermineanawardforEmma’sphysicalandmentalpainandsuffering(pastandfuture)andherlossofcapacityforenjoymentoflife.Thejudgeemphasizesthat“noevidenceofthevalueofintangiblethings,suchasmentalorphysicalpainandsuffering,hasbeenorneedbeintroduced.Youarenottryingtodeterminevalue,butanamount that will fairly compensate the plaintiff for the damages she hassuffered.Thereisnoexactstandardforfixingthecompensationtobeawardedfortheseelementsofdamage.”Youshoulduseyourjudgmenttodecideafairamount.The survey then asks the participant to determine an award for pain and

suffering or punitive damages. In the punitive damages scenario, the survey asks:

“Pleasewritedown thedollaramount thatyouwouldawardAndrew forpunitive

damages,asinstructedbythejudge.”Inthepainandsufferingscenario,thesurvey

asks:“PleasewritedownthedollaramountthatyouwouldawardEmmaforherpain

and suffering (past and future) and her loss of capacity for enjoyment of life, as

instructedbythejudge.”

Finally,onceaparticipantentershisaward,thesurveyasksforanoptional

explanation for the amount awarded, aswell as certain demographic information

(discussedfurtherbelow).68

Fortheactivetreatmentconditions,thestemcontainsafourthparagraphthat

containscertaininformationregardingawardsinpriorcomparablecases.Forconditions

68Note,participantswerenotpermittedtoreturntopreviouspages.Therefore,onceaparticipantsubmittedhisawardandproceededtothefollowingpages(requestinganoptionalexplanationordemographicinformation),hewasnotpermittedtoreturntothepagerequestinganawardamount.

176

involvingthelistforminthepunitivedamagesscenario,theparagraphstates:

Additionally,thejudgeinformsyouthatinfiveprevioussimilarcasesjurieshavedeterminedawardsforpunitivedamagesintheamountsof[$____________,$____________,$____________,$____________,and$____________]. The judge indicatesthatthisinformationregardingpriorawardsisintendedasguidanceonly,andthatyoumayuse(ornotuse)theinformationasyouseeappropriate.

Fortherangeform,theparagraphstates:

Additionally,thejudgeinformsyouthatinprevioussimilarcasesjurieshavedetermined awards for punitive damages in amounts ranging from[$____________ to $____________]. The judge indicates that this informationregardingpriorawardsisintendedasguidanceonly,andthatyoumayuse(ornotuse)theinformationasyouseeappropriate.

Andfortheaverageform,theparagraphstates:

Additionally,thejudgeinformsyouthatinprevioussimilarcasesjurieshavedeterminedawardsforpunitivedamagesinamountsaveraging[$____________].Thejudgeindicatesthatthisinformationregardingpriorawardsisintendedasguidanceonly,andthatyoumayuse(ornotuse)theinformationasyouseeappropriate.

Analogousparagraphsareusedforeachlevelofformforthepainandsuffering

scenario.Further,intheactualsurveys,thebracketedmonetaryvaluesabovearefilledin

withprior-awardvalues,wherethevaluesdependonthetreatmentconditiontowhicha

participantisrandomized—andparticularly,onthelevelsofbiasandvariabilityassociated

withthattreatmentcondition.Thesevaluesarediscussedinthefollowingsubsection.

3.4 DeterminingPrior-AwardInformation

Constructingthesurveystobepresentedtoparticipantsinvolveddetermining

numericalvaluesthatwoulddefineeachtreatmentcondition.Forexample,constructinga

surveycorrespondingtoatreatmentconditioninvolvingunbiasedprior-awardinformation

intheformofanaveragerequiredchoosinganawardvaluetopresenttoparticipantsas

177

theaverageofpriorawards.Webeganbyestimatingthe“correct”awardineachscenario,

definedasinTheLogicofCCG,asthemeanofthedistributionofawardsthatwouldresult

frominfinitelyrepeatedadjudications(ofthecontrolcondition)undervarious

conditions.69Onceweestimatedthe“correct”awardineachscenario,weusedtheestimate

todefine,forexample,whatconstitutesbiasedversusunbiasedprior-awardinformation.

Weestimatedthe“correct”awardsusingthemedianofthecontrolconditionineach

scenario.Foranumberofreasons,themedianislikelytoserveasabetterestimatorthan

themean.Forexample,themeanwouldbetooheavilyinfluencedbyafewextremeoutliers

intherelativelysmallsamplesinthecontrolconditions—relativetotheinfinitelyrepeated

adjudicationsthatdeterminethe“correct”awards.Additionally,theawardsinthisstudy

aredeterminedbymockjurors—notactualjuriesinatrialgovernedbyajudge,allofwhich

servetolimittherightskewofthedistributionsofactualawards.70

Importantly,forpurposesofthecurrentstudy,itisnotnecessarytodefineand

estimatea“correct”award.Ourconclusionswouldnotbeweakenedwerewetoanalyze

thetreatmentconditionswithreferencetotheresultsinthecontrolconditiondirectly,

withoutreferencetoa“correct”award.Forexample,withouttakingapositiononhowto

characterizea“correct”award,wecouldinvestigatetheeffectofaparticulartreatment

conditiononmagnitude,relativetothemagnitudeoftheawardinthecontrolcondition.

Afterall,CCGisintendedtoimproveaccuracybyreducingerrorfromdispersion

substantiallymorethanitaddserrorfromdistortion—whatevertheinitial“distortion”

69Seesupra§2.3.

70Seediscussionregardinglimitationsassociatedwiththepresentstudy,infra§5.

178

maybe.Byreferencinga“correct”award,wesimplyassumethattheawarddetermination

isinitiallyundistorted,orunbiased.

Tobesafe,iftheinitialbiasweresufficientlyextreme,wemay,undercertain

conditions,notwanttoreducevariabilityaroundthecentralaward.Butsuchaproblem

wouldrequireanextremeinitialbias,andthereisnoreasontoassumethatsuchabias

exists(evenif,forourpurposes,therewerecausetoassumesomesourceofbiasinthefirst

place).Furthermore,anyassumptionthatsuchabiasexistedwouldbeinconsistentwith

thepositionoftheSupremeCourt,lowercourts,andmanycommentators,thatreducing

randomvariation,withoutmore(e.g.,withoutcausingbias),isbeneficial.Inanyevent,for

purposesofclarityandsimplicity,aswellasforconsistencywithpreviousliterature,we

definethe“correct”awardsasabove,andestimatethemusingthemediansofthecontrol

groupawards.

Thus,toobtaintheseestimates,weconductedapilotstudydesignedidenticallyto

thecontrolstudydescribedabove.WeusedAmazon’sMechanicalTurktorecruitasample

of400pilotparticipants.71Eachparticipantwasrandomlyassignedtooneofthetwolevels

ofscenario(painandsufferingorpunitivedamages),resultinginasampleof200

participantsforeachscenario.Asinthecontrolconditionsinthemainstudy,pilot

participantswereprovidednoprior-awardinformationandwereaskedtodeterminean

awardbasedonthefactpatterntheyreceived.

71Anadditional200participantsweresurveyedforpurposesofaseparateanalysis.Thedatacollectedforthatanalysisarenotrelevanttothispaper,andthereforearenotdiscussed.

179

Assuggestedabove,participantsrandomizedtoreceiveoneoftheactivetreatment

conditionsinthemainstudywereprovidedwithprior-awardinformationasanumerical

summaryintheformofeitheranaverage,arange,oralistofvalues.Todeterminethese

numericalsummariesforeachscenarioandforeachpossiblecombinationofbiasand

variability,weusedtherawdatafromthepilotstudy.Pilotstudydatawerethenexcluded

fromsubsequentanalyses.

Foreachlevelofscenario,theempiricalmedianofawardamountsinthe

correspondingpilotsampleisusedastheunbiasedaverageandthe30thpercentileof

awardamountsisprovidedasthebiasedaverage.72Additionally,foreachlevelofscenario,

the15th,25th,50th,75th,and85thpercentilesofawardamountsinthecorrespondingpilot

sampleareusedastheunbiasedhighvariabilitylist,andthe40th,45th,50th,55th,and60th

percentilesareusedastheunbiasedlowvariabilitylist.Finally,foreachlevelofscenario,

the15thand85thpercentilesoftheawardamountsfromthecorrespondingpilotsample

areusedastheunbiasedhighvariabilityrange,andthe40thand60thpercentilesareusedas

theunbiasedlowvariabilityrange.

Todefinehighvariabilityandlowvariabilityprior-awardinformationinthebiased

conditions,weapplythefollowingprocedure.First,abiasratioiscalculatedastheratioof

the30thpercentiletothemedianwithineachlevelofscenario.Foreachlevelofscenario,we

thenmultiplythevaluesdeterminedfortheunbiasedhighvariabilitylistandunbiasedlow

variabilitylistbythecorrespondingbiasratiotoconstructthebiasedhighvariabilitylist

72Bythedefinitionof“bias,”anyvalueotherthantheempiricalmedianofthecorrespondingpilotsampledistributioncouldbeusedtodefine“biased”prior-awardinformation.Here,the30thpercentilewaschosentoreflectverysubstantial,butnotentirelyunrealistic,court“error”indeterminingasetofpriorawards.

180

andbiasedlowvariabilitylist,respectively.Weapplythesameproceduretoconstructthe

biasedhighvariabilityrangeandbiasedlowvariabilityrangeforeachlevelofscenario.

TableAprovidesthenumericalprior-awardvaluesthatwereprovidedtoparticipantsin

themainstudyineachoftheactivetreatmentconditions.

TableA.PriorAwardInformationProvidedintheVariousActiveTreatmentConditionsScenario Biasand

VariabilityForm Prior-AwardInformation73

PainandSuffering

UnbiasedLowVariability

Average $2mRange $1.12mto$4.7mList $1.12m,$2m,$2m,$3m,$4.7m

UnbiasedHighVariability

Average $2mRange $200kto$15mList $200k,$500k,$2m,$10m,$15m

BiasedLowVariability

Average $1mRange $560kto$2.35mList $560k,$1m,$1m,$1.5m,$2.35m

BiasedHighVariability

Average $1mRange $100kto$7.5mList $100k,$250k,$1m,$5m,$7.5m

PunitiveDamages

UnbiasedLowVariability

Average $1mRange $500kto$1mList $500k,$500k,$1m,$1m,$1m

UnbiasedHighVariability

Average $1mRange $10kto$10mList $10k,$52.5k,$1m,$5m,$10m

BiasedLowVariability

Average $100kRange $50kto$100kList $50k,$50k,$100k,$100k,$100k

BiasedHighVariability

Average $100kRange $1kto$1mList $1k,$5.25k,$100k,$500k,$1m

73Weuse“m”todenotemillionsand“k”todenotethousands.

181

3.5 AssessingandCorrectingCovariateBalance

Duetoresourceandprivacyconstraints,covariatedatawereunavailablepriorto

thestartoftheexperiment.Inparticular,dataonninecovariates—age,sex,ethnicity,

education,employmentstatus,maritalstatus,householdincome,residentialcommunity

type,andpoliticalaffiliation—werecollecteduponaparticipant’scompletionofthesurvey.

Itisdesirabletohavereasonablecovariatebalanceacrossalltreatmentconditions.

However,withninecovariates,therewasahighprobabilityofobservingimbalanceonat

leastonecovariateamongthe22conditions.Thiswassignificant,becauseimbalancecould

preventusfromknowingwhetheranyobservedeffectswereattributabletothe

interventionortotheimbalance.Therefore,oncethedatawerecollected,andbefore

proceedingtothegeneralanalysisphaseoftheexperiment,wesoughttoexamineand

correctforanysubstantialcovariateimbalanceproducedbytheobservedrandomization.

But,toavoiddata“dredging”andensurethatcovariatebalancewouldbecorrectedinan

objectivemanner,weanalyzedthecovariatedatainaso-called“secondarydesignphase.”

Specifically,weanalyzedthedataintwophases.First,weremovedalloutcomedataand

consideredonlycovariatedata.Priortoaccessingthecovariatedata,however,wedesigned

andfinalizedproceduresforassessingcovariatebalanceandforaddressingany

imbalances.74

Toassesscovariatebalanceacrossthe22treatmentgroups,weexamineddataon

theninedemographicvariablesmeasuredinthesurvey.Additionally,weexaminedtwo

74DesignandAnalysisProtocolonfilewithauthors.Iftheobservedcovariatebalancewerenotsuitable,wewouldhaveproceededbyborrowingmethods,suchas“trimming”basedonpropensityscores,establishedforcausalinferencewithobservationaldata.

182

variablesconstructedusingsurveymetadataregardingenrollmenttimes—timeofdayand

dayofweekofsurveyenrollment—tocheckforsystematicdifferencesinparticipantsbased

onstudyenrollmenttimesthatmayinfluenceoverallcovariatebalanceandvalidateour

assumptionthatparticipantsenrolledinthestudyrandomly.75

Theobservedrandomizationproducedtreatmentgroupsthatwereapproximately

equalinsize,withthesmallesttreatmentgroupcontaining245participantsandthelargest

groupcontaining260participants.Becausethesamplesizeofeachtreatmentgroupwas

approximatelyequalacrossthe22groups,weappliednoweightingbytreatmentgroup

sizewhenassessingcovariatebalance.

WeanalyzedcovariatebalanceusingPearson’schi-squaredtest.When

implementingthistestinpractice,itispreferable,forpurposesofstatisticalvalidity,to

havefiveormore(expected)observationsineachcellofthecovariatecontingencytable.76

Ifthiscriterionisnotsatisfied,itiscommonpracticetocombine,wheresensible,

neighboringlevelsofthetable.Wethusfollowedthisprocedureforlevelsofcovariatesthat

containedlessthan5observationspercell,inordertoensurethateachcellwouldcontain

asufficientnumberofobservations.Inparticular,wecombinedcertainneighboringlevels

75Wealsoconsideredathirdvariablerelatedtoenrollmenttimes—weekend—whichindicateswhethereachparticipantenrolledduringaweekend(definedasanytimeofdayonSaturdayorSunday).Nosignificantimbalancesonthisvariablewereidentified,andbecausethisindicatorisadeterministicfunctionofdayofweek,whichisalsoshowntobebalancedacrossall22treatmentgroups,weexcludedweekendfromfurtheranalyses.

76SeegenerallyWilliamG.Cochran,Theχ2TestofGoodnessofFit,23(3)THEANNALSOFMATHEM.STAT.315(1952).

183

withineachofthefollowingcovariates:age,ethnicity,education,employment,income,and

residentialcommunitytype.77

Independentchi-squaredtestsrevealednostatistically“significant”differences

amongthe22treatmentgroupsinthefrequencydistributionofeachofthenine

demographicvariables(includingage(p=0.15),sex(p=0.55),ethnicity(p=0.82),

education(p=0.33),employment(p=0.87),maritalstatus(p=0.76),income(p=0.34),

residentialcommunitytype(p=0.34),andpoliticalaffiliation(p=0.71)).78Similarly,no

significantdifferencesamongtreatmentgroupswereidentifiedforeitherofthe

enrollment-timevariables(includingdayofweek(p≈1)andtimeofday(p≈1)).The

observedp-valueforthechi-squaredtestoneachoftheenrollment-timevariables

supportsourassumptionthatparticipantsenrolledinthestudyrandomlyandprovides

evidencethattherandomizationschemeperformedasintended.

77Forage,weaggregatedlevelsforagesabove61tocreateone“Over61”level.Forethnicity,participantswhomarked“AmericanIndianorAlaskaNative”wereaggregatedwithparticipantsmarkingthevalue,“Other.”Foreducation,participantswhomarked“None,”“Somehighschool,”or“Highschoolgraduate”wereaggregatedintothelevelof“Highschoolorless,”andparticipantswhomarked“Master’sdegree,”“Doctoratedegree,”“Lawdegree,”or“Medicaldegree”wereaggregatedintothelevel,“Master’sdegreeorhigher.”Foremploymentstatus,participantswhomarked“Retired”wereaggregatedintothelevel,“Other.”Forincome,levels“$100,000-$150,000”and“Over$150,000”wereaggregatedintothelevel,“Over$100,000.”Finally,forresidentialcommunitytype,participantswhomarked“Suburban”or“Don’tknow”wereaggregatedintothelevel“Suburbanoruncertain.”Additionally,forthecovariatesex,weobserved20participantswhomarkedtheresponse“Other”(averagingoneorfewerobservationspertreatmentgroup).Becausethereisnoclearwaytorecodetheseobservationsintoeitherthe“Male”or“Female”levels,weexcludedtheseparticipantsfromsubsequentanalyses.Iftherearesystematicdifferencesinresponsepatternsforthesubpopulationthatidentifiesas“Other,”theexclusionoftheseobservationsmaylimitourabilitytogeneralizeresultsfromthisstudytothissubpopulation.However,weperformedpost-hocanalysestoevaluatetheimpactofexcludingtheseparticipantsfromthestudyandtounderstandthegeneralizabilityofourresults.Oneparticipantwhoidentifiedas“Other”wasremovedinthedatacleaningprocess(describedbelow);andcorrelationanalysisoftheremaining19observationsindicatesnosignificantassociationsbetweenidentificationas“Other”andawardamount,bothwithinandacrosslevelsofscenario.Thissuggeststhatremovalofthese19participantsfromouranalysishasnoconsiderableimplicationsonthegeneralizabilityofthestudy.

78Wepresenthereraw,unadjustedp-values.

184

Tofurtherconfirmsuitablebalanceforrelevantcomparisongroups,weperformed

pairwisecomparisonstoevaluatedifferencesinfrequencydistributionsoneachcovariate

foreachofthe pairsofdistincttreatmentgroupsusingFisher’sexacttest.79To

correctformultiplecomparisons,weappliedtheprocedureproposedbyBenjaminiand

Hochberg(1995),hereinreferredtoasthe“Benjamini-Hochbergprocedure,”tocontrolthe

falsediscoveryrate(FDR).80Afteradjustmentformultiplecomparisons,nosignificant

differencesbetweenpairsoftreatmentgroupswereidentifiedonanyoftheeleven

variablestested.

Wealsoperformedtestsfortwo-wayinteractionsamongtheninedemographic

variablesandthetwoenrollment-timevariables.Theseinteractioneffectscanhelpusto

betterunderstandtheobservedsampleandevaluatehowinferencefromthisstudy

generalizestothepopulationofinterest.Again,weemployedtheBenjamini-Hochberg

proceduretoadjustformultiplecomparisons.Afteradjustment,ofthe tests

performed,46interactionswereidentifiedasstatistically“significant”attheoveralllevelof

α=0.05.81Suchcorrelationsarenotsurprising(forexample,agemaycorrelatewith

experienceand/orresponsibility,andthuswithemployment;dayofweekmaycorrelate

79Wealsoperformedcomparisonsacrossaggregationsoftreatmentgroupscorrespondingtothe60effectsofinterestdefinedinourprimaryanalysisandfoundsuitablebalanceforeachaggregation.

80YoavBenjamini&YosiHochberg,ControllingtheFalseDiscoveryRate:APracticalandPowerfulApproachtoMultipleTesting,57(1)J.OFTHEROYALSTAT.SOC.SERIESB(STAT.METHODOLOGY)289.

81Forexample,sexsignificantlycorrelateswithmaritalstatus,agesignificantlycorrelateswithemploymentstatus,anddayofweeksignificantlycorrelateswithtimeofday.

222

!

"#

$

%&= 231

112

!

"#

$

%&= 55

185

withlighterworkschedulesandthuswithtimeofdayofsurveyenrollment).Ofgreater

interestaretheinteractionsofthedemographicvariableswiththeenrollment-time

variables,sinceextremecorrelationsofthistypemayhaveimplicationsregardingthe

assignmentmechanism.82Inthesetests,significantinteractionswereidentifiedfortimeof

dayandthedemographicvariablessex,income,andmaritalstatus.However,becausewe

observedsuitablebalanceoneachoftheelevenvariablesacrosseachofthe22treatment

groups,therewasnoreasontobelievethattheidentifiedinteractionsmayhaveintroduced

confoundingintotherandomizationorhavenegativeimplicationswithrespecttothe

validityofthestudy.Theseinteractioneffectssimplyservetocharacterizethepopulation

representedbyoursampleandmaybefurtherconsideredinpost-hocanalysesand/or

whenattemptingtogeneralizetheresultsofthestudy.

Insummary,ouranalysisofthecovariatedataindicatedthatsuitablebalanceonall

demographicvariablesandenrollment-timevariablesacrossthe22treatmentgroupswas

achievedbytheobservedrandomization.Therefore,weproceededtotheanalysisphaseof

theexperimentwithoutapplyinganycorrectionsoradjustmentstoimprovebalance.

3.6 CausalInferenceforFactorialEffects

Theanalysisofdatafromfactorialexperimentsoftenreliesonageneralizedlinear

modelframework(i.e.,analysisofvariance(ANOVA)).However,asdiscussedin2K

FactorialDesigns,theseapproacheshavedrawbacksthatcanimpedetheabilitytomake

82Forexample,iffemalesenrolledinthestudyexclusivelyonMondays,thenumberofrandomizationsthatarepossibletoobserveusingourrandomizationschemewoulddecrease,havingimplicationsforouranalysisforpurposesofmakingvalidinferences.

186

causalconclusionsabouttheexperimentalfactors.83Wethereforebaseouranalysesand

estimationofcausaleffectsonthepotentialoutcomesframeworkofNeyman,84often

referredtoastheRubinCausalModel(RCM).85Wefollowthebasicnotationand

philosophyofestimationin2KFactorialDesigns,whichdevelopedatheoreticalframework

forcausalinferencefromfactorialdesignsusingthepotentialoutcomesmodel.

UndertheRCM,eachunit,i.e.,eachparticipant,inthisexperimenthas22“potential

outcomes,”oneforeachpossibletreatmentcombination.Forexample,aparticipantmay

haveawarded$4millionhadhebeenrandomizedtothepunitivedamagesscenarioofthe

controlcondition,$2millionhadhebeenrandomizedtotheunbiased,lowvariability,

average,punitivedamagestreatmentcondition,$1millionhadhebeenrandomizedtothe

biased,lowvariability,average,punitivedamagestreatmentcondition,andsoonandso

forthforall22possibletreatmentcombinations.TheRCMframescausalinferenceasa

missing-dataproblem:becausewecanobserveonlyonepotentialoutcomeforeachunit—

theonetowhichtheunitwasinfactassigned—wedonotknow,andthereforemust

estimate,thevaluesoftheunobservedpotentialoutcomestomakecausalconclusions.86

83Forexample,onedrawbackofthelinearmodelframework,assuggestedin2KFactorialDesigns,istherequirementthatthecausalestimandsbedefinedasparametersoftheprobabilitydistributionoftheobservedresponse.Tothecontrary,ourresultsdonotrelyondistributionalassumptions.See2KFactorialDesigns.

84SeeJerzyNeyman,OntheApplicationofProbabilityTheorytoAgriculturalExperiments.EssayonPrinciples.Section9.,RocznikiNaukRolniczychTomX[inPolish]:translatedin5StatisticalScience465(1923).

85SeePaulW.Holland,StatisticsandCausalInference,81(396)J.oftheAm.Stat.Ass’n.945(1986).

86Seegenerallyid.;Rubin,supranote60;2KFactorialDesigns.

187

Ingeneral,afactorial“maineffect”ofanexperimentalfactorforeachunitisthe

differenceinafunction(e.g.,themean)ofthepotentialoutcomesbetweenasubsetofthe

potentialoutcomescorrespondingtoonelevelofthefactorandtheremaining(or,herein,a

subsetof)potentialoutcomes.87Forexample,theunit-levelmaineffectofunbiasedprior-

awardinformationrelativetobiasedpriorawardinformationisthedifferencebetweena

function(suchasthemean)ofthepotentialoutcomesassociatedwiththesubsetof

treatmentcombinationsthatincludeunbiasedprior-awardinformation(thatis,the

outcomesthatwouldoccurhadtheunitbeenassignedtothesetreatmentcombinations)

andthesamefunctionofthepotentialoutcomesassociatedwiththesubsetoftreatment

combinationsthatincludebiasedprior-awardinformation(thatis,theoutcomesthat

wouldoccurhadtheunitbeenassignedtothesetreatmentcombinations).

87Notationally,asin2KFactorialDesigns,letKdenotethenumberofexperimentalfactors,whichareindexedbyk,eachwithpklevels,suchthatthereare treatmentcombinations.Letzj,forj=1,…J,denoteoneparticulartreatmentcombination,definedbyaK-dimensionalvector,wherethekthelementindicatesthelevelofthekthfactor,fork=1,…,K.LetZdenotethesetofallpossibletreatmentcombinations.Further,letNdenotethetotalsamplesize,indexedbyi=1,…,N.AndletYi(z)denotethepotentialoutcomeoftheithunitunderexposuretotreatmentz.2KFactorialDesignsat730.LetYi=(Yi(1),…,Yi(J))bethevectorofallpotentialoutcomesforuniti.DefinegastheJ-1vectors,indexedbyj=1,…,J-1,suchthatgjisavectorofdimensionJwithvaluesthatindicatethelevelsoffactorscorrespondingtopotentialoutcomes.Id.Forexample,ifwerefertoscenarioasfactor1,theng1isavectorofeleven-1’sfollowedbyeleven+1’s,indicatingthatthefirsteleventreatmentcombinationsz1,…,z11havethescenariofactorsettopainandsufferingandthefollowingelevencombinationshavethescenariofactorsettopunitivedamages.Usingthisnotation,thefactorialmaineffectoftreatmentcombinationjcomparedtoallothertreatmentcombinationsforuniticanbedefinedgenerallyas

τ ij =1

#{gj =1}Yi

gj=1∑ −

1#{gj ≠1}

Yigj≠1∑ where #{gj =1} denotesthenumberofelementsofgjthatare

equaltoone.Similarly,thefactorialeffectoftreatmentcombinationjrelativetotreatmentcombinationj*for

uniticanbedefinedasτ ij, j*=

1#{gj =1}

Yi −gj=1∑ 1

#{gj*=1}

Yigj*=1∑ .Factorialeffectsforeachlevelpkofeach

factorkaredefinedbyaggregatingovertreatmentcombinationsthathavefactorksettothatlevel.Thesedefinitionsaresimilarlyextendedtosubsetsoftreatmentcombinations.Seeid.

J = p1 ×...× pk

188

Two-way,three-way,andfour-way“interactioneffects”canbedefinedsimilarly.For

example,theunit-leveleffectofunbiasedhighvariabilityprior-awardinformation,relative

tounbiasedlowvariabilityprior-awardinformation,canbedefinedasthedifference

betweenthemeanofthepotentialoutcomesassociatedwiththesubsetoftreatment

combinationsthatincludeunbiasedhighvariabilityprior-awardinformationandthemean

ofthepotentialoutcomesassociatedwiththesubsetoftreatmentcombinationsthat

includeunbiasedlowvariabilityprior-awardinformation.88

Notethatwedonottest“across”levelsofscenario.Thatis,forpurposesofanalysis,

wetreatthepainandsufferingscenarioandthepunitivedamagesscenarioasseparate

studies;andallcomparisons,correspondingtoeffectsofinterest,aremadewithinlevelsof

scenario—thatis,withinthepainandsufferingscenarioorthepunitivedamagesscenario.

Thus,wesometimesreferto,e.g.,theeffectofbiasorvariability,withinalevelofscenario,

asa“maineffect.”

3.7 EstimandsandEstimators

Theprimaryobjectivesoftheexperimentaretoassessthefinite-populationeffects

ofprior-awardinformation,fordifferentlevelsofbias,variability,andform,onthe

magnitude,spread,andaccuracyofresultingawardvalues,whereaccuracyisdefinedasthe

88Throughoutthestudy,wemakethestableunittreatmentvalueassumption(SUTVA).Thismeans1)thatthepotentialoutcomeofaunitdependsonlyonitsownassignment,andnotontheassignmentsofotherunits,and2)thatthereareno“hiddenversions”oftreatment.2KFactorialDesignsat730.Forthisexperiment,wehavenoreasontobelievethattherewasinterferenceacrossunitsbecausethepopulationfromwhichwerecruitedparticipantsconsistsofmanydistinctusersfromacrosstheUnitedStates.Further,notethatMechanicalTurkrequiredthateachsurveyparticipanthaveauniqueIPaddress.Similarly,becausewecarefullycontrolledtheadministrationofallsurveys,wearearguablyjustifiedinassumingthattherewerenohiddenversionsoftreatment.Weassumethateachparticipantreadthesurveytheywereassignedinitsentirety.

189

proximityoftheawardstothe“correct”awardintermsofbothbiasandvariance.These

objectivesmotivateourchoiceof“estimands,”thequantitiesweareinterestedin

estimating.Wedefinethreegeneralestimands—oneformagnitude,oneforspread,andone

foraccuracy—andcorresponding“estimators,”functionsofthedatathatweuseto

estimatethequantitiesofinterest,i.e.,theestimands.

Magnitudecanbecapturedquantitativelyusingameasureofcentraltendency,such

asthemeanormedian.Becausetheoutcomeismonetaryandknowntobepositively

skewed,forpurposesofanalysiswedefinemagnitudeasthemeanofthelogarithm-

transformedawardamounts.Althoughthisdefinitioninvolvesanumberofdrawbacks,

discussedbelow,itreducestheeffectoftherightskewanditallowsustocapturerelative,

ratherthanabsolute,differences.89Ontheotherhand,wedefinespreadastheinterquartile

range(IQR)oftheoutcome,whichismorerobusttooutliersthanthevariance;andwe

defineaccuracyastheinverseofthemean-squarederror(MSE),whichisequaltothesum

ofthevarianceandthesquaredbias.Thus,tocalculateaccuracy,weestablishanumerical

rulefordefiningthe“correct”award,reflectingourestimatorforthe“correct”award

discussedsupra§3.4.Specifically,withineachlevelofscenario,themedianamount

awardedinthecontrolgroupofthemainstudyisusedasthe“correct”award.

Wethusdefinethefinite-populationfactorialeffectofeachexperimentalfactorand

interactionsbetweenfactorsonthemagnitudeofawardvaluesasthedifferencebetween

89Forexample,thedifferencebetween$0and$100shouldarguablybeinterpreteddifferentlythanthedifferencebetween$1,000,000and$1,000,100.Althoughcapturingrelativedifferencesisimportant,thisdefinitionmaynotbeidealorappropriateinthepresentcontext.Wediscussthis,andalternativedefinitions,infra§4.4.

190

themeanofthelogarithm-transformedpotentialoutcomesassociatedwitheachofthe

comparisongroups,respectively.Weapplythisdefinitiontoallsuchcomparisongroups,

whetherformaineffects,ortwo-way,three-way,orfour-wayinteractioneffects.For

example,inthepainandsufferingscenario,theeffectonmagnitudeoflowvariability

unbiasedprior-awardinformation,relativetohighvariabilityunbiasedprior-award

information,isthedifferencebetweenthemeanofthelogarithm-transformedpotential

outcomesassociatedwiththeformerandthemeanofthelogarithm-transformedpotential

outcomesassociatedwiththelatter.

Weestimatetheseeffects(sincetheestimandisafunctionofunknownquantities)

byusingthecorrespondingsetsofobserveddata.90Wedefinethefinite-populationfactorial

effectonthespreadofawardvaluesastheinterquartilerange(IQR)—definedasthe

differencebetweenthe75thpercentileandthe25thpercentile—ofthepotentialoutcomes

associatedwitheachofthecomparisongroups,respectively.Weestimatethiseffectusing

thecorrespondingsetsofobserveddata.91

90Moreformally,wedefinethefinite-populationfactorialeffectoftreatmentcombinationjcomparedtoall

othertreatmentcombinationsonthemagnitudeofawardvaluesasτ mag⋅ j =

1#{gj =1}

!Ygj=1∑ −

1#{gj ≠1}

!Ygj≠1∑ ,

where denotesthemeanofthelogarithm-transformedpotentialoutcomes.We

estimatethisquantityusingtheestimator τ̂ mag⋅ j =

1#{gj =1}

!Y obs

gj=1∑ −

1#{gj ≠1}

!Y obs

gj≠1∑ .Asabove,these

definitions,aswellasthedefinitionsforspreadandaccuracybelow,aresimilarlyextendedtocomparisonsofparticulartreatmentcombinationsorsubsetsoftreatmentcombinationstootherparticulartreatmentcombinationsorsubsetsoftreatmentcombinations.Seesupranote87.Seegenerally2KFactorialDesigns.

91Wedefinethefinite-populationfactorialeffectoftreatmentcombinationjcomparedtoallothertreatmentcombinationsonthespreadofawardvaluesas

!Y =1N

log(Yi +1)i=1

N

∑

191

Finally,wedefinethefinitepopulationfactorialeffectontheerrorofaward

amounts,theinverseoftheaccuracyofawardamounts—andthusontheaccuracyof

awardamountsitself—asthedifferencebetweentheMSEs,definedasthemeanofthe

squareddifferencesbetweeneachvalueandthe“correct”value,ofthepotentialoutcomes

associatedwitheachcomparisongroup,respectively.Here,again,weestimatetheeffecton

accuracyusingtheobserveddatacorrespondingtothecomparisongroups.92

Usingthefactorial-effectestimatorsweestimateaneffectbycomputingthevalues

ofmagnitude,spread,oraccuracyforeachofthetreatmentcombinationsinvolvingthe

factor(orcombinationoffactors)ofinterest,andthenseparatelyaveragingovereachof

τ spread⋅ j =

1#{gj =1}

(q0.75(Y )− q0.25(Y ))−gj=1∑ 1

#{gj ≠1}(q0.75(Y )− q0.25(Y ))

gj≠1∑ ,where denotesthesth

quantileoftheempiricaldistributionof .Thiseffectisestimatedusingtheestimator

τ̂ spread⋅ j =

1#{gj =1}

(q0.75(Y obs )− q0.25(Y obs ))−gj=1∑ 1

#{gj ≠1}(q0.75(Y obs )− q0.25(Y obs ))

gj≠1∑ .Seegenerally2K

FactorialDesigns.

92Wedefinethefinite-populationfactorialeffectoftreatmentcombinationjcomparedtoallothertreatment

combinationsontheaccuracyofawardamountsasτ acc⋅ j =

1#{gj =1}

(Y −α)2gj=1∑ −

1#{gj ≠1}

(Y −α)2gj≠1∑ ,

whereαisthe“correct”award,whichtakesonadifferentvalueforeachscenario.Ifz1representsthecontrolconditioninthepainandsufferingscenarioandz12representsthecontrolconditioninthepunitivedamagesscenario,wehave ,astheestimatorofthe“correct”awardforthepainandsuffering

scenarioand astheestimatorofthe“correct”awardforthepunitivedamagesscenario.Wethenestimatethefactorialeffectoftreatmentcombinationjontheaccuracyofawardamountsforthepainandsufferinglevelofscenariousingtheestimator

τ̂ acc,PS⋅ j =

1#{gPS

j =1}(Y obs −α̂ PS )2

gj=1∑ −

1#{gPS

j ≠1}(Y obs −α̂ PS )2

gj≠1∑ ,andforthepunitivedamageslevelof

scenariousingtheestimator τ̂ acc,PD⋅ j =

1#{gPD

j =1}(Y obs −α̂ PD )2

gj=1∑ −

1#{gPD

j ≠1}(Y obs −α̂ PD )2

gj≠1∑ ,wheregjPSand

gjPDarevectorswithvaluesindicatingthelevelofeachfactorforeachpotentialoutcomewhenthescenariofactorissettopainandsufferingorpunitivedamages,respectively.Seegenerallyid.

qs (Y )

Y

α̂ PS =med(Yiobs (z1))

α̂ PD =med(Yiobs (z12 ))

192

thetreatmentcombinationswiththesamelevelofthefactor(orcombinationoffactors).

Forexample,forthemaineffectofbiasonmagnitudewithinthepainandsufferingscenario,

we:1)calculatethemagnitudeofawardamountsseparatelyforeachofthe10treatment

combinationsreceivingeitherbiasedorunbiasedpriorawardinformationinthepainand

sufferingscenariousingthemeanofthelogarithm-transformedvalues;2)averagethe

valuesfromstep(1)acrossthe5treatmentcombinationsreceivingbiasedinformation,and

separatelyaveragethevaluesfromstep(1)acrossthe5treatmentcombinationsreceiving

unbiasedinformation;and3)calculatethedifferenceoftheaveragesobtainedinstep(2).

Alsonotethattheestimandsandestimatorsdefinedinthissubsectionapplytomain

effects,aswellasalltwo-way,three-way,andfour-wayinteractioneffects.

3.8 AnalysisProcedure

Oncewecalculatetheestimatesofthefactorialeffectsusingobserveddata,weare

interestedinunderstandingthe“statisticalsignificance”oftheestimatedeffects,whichcan

beinterpretedasanindicatorofhowunlikelytheobserveddifferencesbetweentreatment

groupswouldbeundercertainhypothesizedeffects.93TheRCMallowssuchinference

usingtheNeymanian(1923,1990)perspective,Fisherianperspective(1925,1935),orthe

Bayesianperspective.94

Becausetheestimatorforcapturing,e.g.,thespreadofawardvaluesisnotan

unbiasedestimatorofitscorrespondingestimand,andforotherreasons,wedecidednotto

93Rubin,supranote60.

94SeeDonaldB.Rubin,BayesianInferenceforCausalEffects:TheRoleofRandomization,6(1)THEANN.OFSTATIST.34(1978).

193

useNeyman’smethod.Instead,weadopttheFisherianapproach,anduserandomization

teststoevaluateFisher’ssharpnullhypothesisof“nounit-leveltreatmenteffects.”For

example,totestwhethertheestimateofthemaineffectofeachfactoronmagnitude,

spread,andaccuracyofawardvaluesis“significant,”weassumeFisher’s“sharpnull”—a

hypothesizedeffect,suchasa“sharpnull”of“zerotreatmenteffect”—inordertoimpute

themissingpotentialoutcomesforeachunitandrepeatedlycalculatethevalueofa

specifiedteststatisticunderdifferentpossibletreatmentassignments.Then,giventhe

distributionofeachteststatistic,wecalculateapproximateFisher“exactp-values”(FEPs)

toindicatethestatistical“significance”ofeffects.Wecanalsousethisprocedureto

constructFisherexactintervalsforeachestimate.

OneofthemajoradvantagesoftheFisherianapproachtoinference,asopposedtoa

model-basedorBayesianapproach,isthatitdoesnotrequireanyassumptionsaboutthe

underlyingdistributionofthedata.Thus,randomizationtestscanbeconstructedand

appliedforanyteststatistic,regardlesswhetheritsdistributionisknown.

Duetolimitedcomputationalresources,wefollowthecommonapproachof

constructingrandomizationdistributionsusingsamplesofthepossiblerandomizations,

ratherthancomputingallpossiblerandomizationsoftheN=5480participantsintothe22

treatmentconditions.

Duetothenumberofexperimentalfactorsandquestionsofinterestinthisstudy,to

estimateandevaluatethestatistical“significance”ofeachoftheeffectsofinterestrequires

numeroushypothesistests.Whentestingmanyhypotheses,theprobabilityofidentifying

falseeffectsincreaseswiththenumberoftestsperformed.Toadjustforthisincreased

194

probabilityoffalseidentifications,itisappropriatetoapplyanadjustment,or“correction,”

toallestimatedp-values.95However,applicationofsuchanadjustmentacrossalarge

numberoftestsgenerallydecreasesthepowerofthestudytoidentifyresultsthatare

actually“significant.”

Toaddressthisissue,wedivideourtestsintoaprimaryanalysiscategoryanda

secondaryanalysiscategory,correspondingtotheeffectsofprimaryandsecondary

interest,respectively,andapplycorrectionsformultiplecomparisonsseparatelywithin

eachcategory.Thetestsintheprimarycategoryrelatetothestudy’sprimaryquestionsof

interest.Thetestsinthesecondarycategoryrelateto1)questionsthatareofsecondary

interest,2)effectsofinterestthatmayprovideconfirmatoryorsupportiveevidenceof

resultsidentifiedintheprimaryanalysiscategory,orthatmayanswerquestionsthat

followfromresultsidentifiedintheprimaryanalysiscategory,or3)exploratoryanalysis.

Becausewearemostinterestedinunderstandingprimaryeffectswithineachlevel

ofscenario,intheprimaryanalysiscategorywetreatasindividualexperiments1)all

treatmentconditionsreceivingthepainandsufferinglevelofscenario,and2)alltreatment

conditionsreceivingthepunitivedamageslevelofscenario.Thus,intheprimaryanalysis

category,wetesteacheffectofinterestwithineachofthetwolevelsofscenarioandapplya

correctionformultiplecomparisonsseparatelywithineach.Inparticular,wetestten

effectswithineachofthetwolevelsofscenariooneachofthethreeoutcomesofinterest

(magnitude,spread,andaccuracy),foratotalof10x2x3=60primaryhypothesistests.96

95SeeBenjamini&Hochberg,supranote80.

96Intheprimaryanalysiscategory,wetestthefollowingtentestswithineachlevelofscenario:1)theeffectofunbiasedprior-awardinformationrelativetonoprior-awardinformation(control);2)theeffectof

195

Alladditionalhypothesistestsareperformedinthesecondaryanalysiscategory,

includingtestsforthemaineffectsofformandvariability,andadditionaltwo-way,three-

way,andfour-wayinteractioneffectsamongbias,variability,andform.Inbothanalysis

categories,wecorrectformultiplecomparisonsaswedidintheSecondaryDesignPhasein

theanalysisphaseofthisstudy,usingtheBenjamini-Hochbergproceduretocontrolthe

false-discoveryrate(FDR).97

Ourprocedureforestimatingfactorialeffectsandcarryingoutinferenceforeach,

whichincludescomputingp-valuestotestthenullhypothesesof“notreatmenteffect”and

constructingintervalsfortheestimatedeffects,isoutlinedbelow.

Estimatingeffects.Foreachestimandofinterest,weusetheobserveddatato

computethevalueofthecorrespondingestimator.Becausetheestimandsaredefinedas

differencesbetweentreatmentgroups,foreachestimate,weareinterestedintestingthe

nullhypothesisof“nodifferencebetweengroups”—thatis,weareinterestedintestingthe

unbiasedprior-awardinformationrelativetonoprior-awardinformation(control)whenprior-awardinformationislowvariability;3)theeffectofunbiasedprior-awardinformationrelativetonoprior-awardinformation(control)whenprior-awardinformationishighvariability;4)theeffectofbiasedprior-awardinformationrelativetonoprior-awardinformation(control);5)theeffectofbiasedprior-awardinformationrelativetonoprior-awardinformation(control)whenprior-awardinformationislowvariability;6)theeffectofbiasedprior-awardinformationrelativetonoprior-awardinformation(control)whenprior-awardinformationishighvariability;7)theeffectofbiasedprior-awardinformationrelativetounbiasedprior-awardinformationwhenprior-awardinformationishighvariability;8)theeffectofbiasedprior-awardinformationrelativetounbiasedprior-awardinformationwhenprior-awardinformationislowvariability;9)theeffectoflowvariabilityprior-awardinformationrelativetohighvariabilityprior-awardinformationwhenprior-awardinformationisunbiased;10)theeffectoflowvariabilityprior-awardinformationrelativetohighvariabilityprior-awardinformationwhenprior-awardinformationisbiased.

97Thisprocedurelimitstheexpectedproportionoftestsfalselyidentifiedas“significant”relativetothetotalnumberoftestsidentifiedas“significant.”Itisawell-acceptedapproachtocorrectingformultiplecomparisonswhileoftenresultinginmorepowerthanmethodscontrollingthefamilywiseerrorrate(FWER),suchastheBonferronicorrectionandtheensembleadjustedp-valueapproach.SeeMariaT.Kimeletal.,TheFalseDiscoveryRateforMultipleTestinginFactorialExperiments,50(1)TECHNOMETRICS32(2012).

196

hypothesisthatthetruedifferencebetweengroupsiszero.Therefore,foreachhypothesis

test,weusethecorrespondingestimatortodefinetheteststatisticforourrandomization

test.

Constructingrandomizationdistributions.Foreacheffectofinterest,weassume

thesharpnullhypothesisof“nounit-leveltreatmenteffect”toimputemissingpotential

outcomesforeachparticipant.WethengenerateasampleofNsim=250,000possible

randomizationsoftheNparticipants.98Foreachofthese250,000possiblerandomizations,

werecomputethevalueoftheteststatisticunderthe(hypothetical)randomization.The

resultingdistributionisreferredtoasthe“randomizationdistribution”oftheteststatistic.

Testinghypotheses.WeapproximateFEPsforeachhypothesisbycalculatingthe

proportionofvaluesintherandomizationdistributionthatwereequaltoormoreextreme

thantheobservedteststatistic.Allestimatedp-valuesareadjustedformultiple

comparisonsusingtheBenjamini-HochbergproceduretocontroltheFDR.Aftercorrection,

weconsiderestimatedp-valuesoflessthanα=0.05tobe“statisticallysignificant.”

Significantp-valuessuggestthatthereissubstantialevidencewithinthedataagainstthe

nullhypothesisofnotreatmenteffect.

Constructingintervals.Foreachfactorialeffect,wecangeneratea95%Fisher

intervalbycalculatingasequenceofraw,unadjustedp-valuescorrespondingtoanull

hypothesisotherthanthehypothesisof“zerotreatmenteffect,”andthenidentifythe

98WechooseNsim=250,000toensureprecisionoftheestimatedp-value.ProfessorsImbensandRubinshowthat,forasampleofthissize,thestandarderrorofeachestimatedp-valuewillbelessthan0.001.SeeGuidoW.Imbens&DonaldB.Rubin,CausalInferenceforStatistics,Social,andBiomedicalSciences(2015).

197

hypothesizedvaluesthatresultinp-valuesequaltoorlargerthan0.05.Forallestimates

weassumeaconstantadditivetreatmenteffect.99

4. Results

Beforediscussingourresults,itisimportanttonotethat,priortoaccessingany

data,wedraftedandfinalizedanExperimentalDesignandAnalysisProtocolthatdetailed

thedesignofthestudyandallaspectsofourintendedanalysis.100Weensuredthatwe

wouldnothaveaccesstothedatabyaskingathird-partyadministratortolockouraccess

ontherelevantdatabase,andtounlockthedata,wheninstructed,onlyuponour

circulation(andpostingtoSSRN)ofafinalizedExperimentalDesignandAnalysisProtocol.

Inthissection,wereportandinterpretourresults.Forpurposesofclarityandease

ofinterpretation,wedividethesectionintotopics,reflectingcategoriesofeffectsof

interest.Foreachtopic,wereportourresultsandaccompanyinginterpretation.Then,in

thefollowingsection,theConclusion,wetieourfindingstogether,relatethemtothemodel

introducedinTheLogicofCCG,anddiscusslimitations.

Asapreliminarymatter,itisimportanttorealizethatouranalysisinvolved

approximately250hypothesistests,reflectingmaineffectsandinteractionsamongthe

variousfactors.Therecanbenumerousteststhatspeaktoaparticulareffect,andsome

resultsmayseemunsupportiveof,oreveninconsistentwith,others.Ourgeneralapproach

tointerpretingthedataisasfollows:First,weexaminethecomparisonthatmostdirectly

99Note,becauseweusethelogarithmtransformationforestimatingfactorialeffectsonmagnitude,weinterprettheresultingintervalsfortheseestimatesasmultiplicativeeffects.

100Onfilewiththeauthors.

198

speakstoaneffectofinterest.Second,weexamineothercomparisonsthatspeaktothe

effectlessdirectly.Third,ifwefindsupportfortheeffectinstep(1)andsupportforthe

effectinstep(2),wegenerallyinterpretthedataasevidencingastrongeffect.Ifwefind

supportfortheeffectinstep(1)butanabsenceofevidencefororagainsttheeffectinstep

(2),wegenerallyinterpretthedataasevidencinganeffect,unlessthereisaparticular

reasontobelieveotherwise.Ifwefindsupportfortheeffectinstep(1)butevidence

againsttheeffectinstep(2),weexaminetheinconsistencythataroseinstep(2).Ifour

examinationrevealsastraightforwardexplanationthatdefusestheinconsistency,we

interpretthedataasevidencinganeffect.Iftheexaminationfailstoreveala

straightforwardexplanation,weinterpretthedataasnotevidencinganeffect;andwe

examinetheresultsforanalternativeexplanation.

4.1 TheData

Thedatacanbedividedintotwodatasets:oneforthepunitivedamagesscenarioand

oneforthepainandsufferingscenario.101Thereareapproximately240-250observationsin

eachtreatmentlevelforeachscenario.Theawarddeterminationsareextremelyvariable.

Awardsrangefrom$0to$15billioninthepunitivedamagesscenario,witha99thpercentile

101Priortoanalyzingthedata,weappliedaninitial“datacleaning”proceduretoensuredataquality.First,toensurethevalidityofresponses(e.g.,thattheyoriginatedfromregisteredMechanicalTurkuserswhometourinclusioncriteria),weexcluded23participantswhoenteredanincorrectpaymentcodethatwasintendedtoindicate,forpurposesofreceivingpayment,thattheparticipantcompletedthesurvey.Second,weexcludedfourparticipantswhoprovidedawardamountsthatweredeemed“nonsensical”byoursoftware.Becauseonlyaverysmallnumberofparticipantswereexcludedfromtheinitialsampleof5,500participants,andtherewasnoevidencesuggestinganyresultingsystematicdistortion,weexcludedtheseparticipantswithoutapplyingadvancedmissingdatatechniques.Additionally,asdiscussedintheMethodologySection,participantswhomarked“Other”fortheirsexwereexcluded,duetoourinabilitytosensiblymergethatcategorywithoneoftheothercategoriesofsexforpurposesoftestingforcovariatebalance.Afterenforcingtheseexclusioncriteria,ourfinalsamplesizeisN=5,458.

199

of$50million,and$0to$1Quadrillion(1×1015 )inthepainandsufferingscenario,witha

99thpercentileof$100million.

Toaddressextremeoutliers,we“truncate”thedatainthepainandsufferingdataset

(inwhichtheproblemisparticularlyacute)atthe99thpercentile.Thismeansthatany

awardsabove$100millionarerecodedto$100million.Thisisimportant,sinceextreme

outlierscancausespuriousresultsthataredominatedbychanceratherthantrueeffects.102

Weappliedthisapproachtothepunitivedamagesdataalso,butfindthatthetruncation

methodinthatscenariohasnoeffectonourfindings,becausetheoutliersinthepunitive

damagesscenarioarefarlessextremeinsizeandeffect.

Thus,unlessstatedotherwise,theresultswereportandinterpretbelowreflectthe

rawdatasetforthepunitivedamagesscenarioandthetruncateddatasetforthepainand

sufferingscenario.Wesometimesusethetruncateddataforthepunitivedamagesscenario

ortherawdataforthepainandsufferingscenario,butonlyforillustrativepurposes—i.e.,

tomakeapointregardingthosedatainparticularor,incertaininstances,forpurposesof

creatingeasily-interpretabledisplays.Insuchinstances,westateclearlythatweareusing

thesealternativedatasets.Below,wealsoexaminetheeffectsofprior-awardinformation

102Notethat,indesigningtheexperiment,weweresensitivetothefactthat,whilemanystudiesuseliberalmethodsforaddressingoutliers,extremevaluesareofparticularinterestinthecurrentstudy.Asaresult,weattemptedtoavoidtheuseofsuchtechniques.However,becausethestudyinvolvesmonetaryvaluesthatparticipantschosefromtheirimagination,theoutliersinthisstudy—andparticularlythoseinthepainandsufferingscenario—wereparticularlyextreme,exceedingevenourexpectations.Uponobservingthemagnitudeoftheawards,weunderstoodthatnotaddressingtheoutliersmayresultinspuriouseffectsandotheranomaliesthatwouldresultfromrandomnessratherthanatrueeffect.Wethereforedecidedtoaddressoutliersinthepainandsufferingscenariousingtheconservativemethoddescribedinthetext.Weproceedusingthetruncateddataset.Notethatsuchextremeawards(andlikelysomethatarefarlessextreme)arehighlyunlikelyinpractice,andwould,inanyevent,beaddressedbythecourtsusingthedeviceofremittitur.

200

onoutliersseparately,aswellastherobustnessofourresultsusinganalternative

truncationthreshold.103Summarystatisticsforeachscenario(rawandtruncated)are

providedinTableB.

TableB.SummaryStatisticsforSample(N=5,458)

4.2 EffectofPrior-AwardInformationonAccuracy

Thedataprovidestrongevidencethatprior-awardinformationreduceserrorand

improvesaccuracy.Inbothlevelsofscenario,andacrossalllevelsofbiasandvariability,

andtheirinteractions,prior-awardinformationhasasignificantpositiveeffectonaccuracy.

FigureV.1illustratestheapproximaterandomizationdistributionsandobservedtest

statisticsforthehypothesistestsof“notreatmenteffect”ineachscenario;andeffectson

error(theinverseofaccuracy)withineachlevelofscenarioaresummarizedinTableCand

103Seeinfra§4.6.

104Weexcludefromthissummaryofrawpainandsufferingdatathreeextremevaluesthataregreaterthanorequalto$100billion.Theseobservationsareremovedfromthissummarysolelyfordescriptivepurposes.

StatisticPunitiveDamages(Raw)

PunitiveDamages

(Truncatedat99thpercentile($50million))

PainandSuffering(Raw104)

PainandSuffering

(Truncatedat99thpercentile($100million))

SampleSize 2751 2751 2704 2707

Mean(millions) $18.3 $2.3 $23.8 $6.7

Median(millions)

$0.5 $0.5 $3.0 $3.0

StandardDeviation(millions)

$397.3 $6.4 $396.0 $14.0

IQR(millions) $0.9 $0.9 $3.5 $3.5

201

FigureV.2below.Thedatasetforthepunitivedamagesscenario,inadditiontothedataset

forthepainandsufferingscenario,istruncatedinthissubsectionforpurposesof

interpretabilityandillustration(althoughourfindingswouldnotchangewerewetouse

thedatawithouttruncation).InTableC,Fisherintervalsareprovidedforthedifferencein

errorbetweeneachtreatmentlevelandcontrol,wheretheMSEinthecontrolgroupforthe

punitivedamagesscenariois$146trillionandtheMSEinthecontrolgroupforthepainand

sufferingscenariois$492trillion.

FigureV.1.Randomizationdistributionforeffectoftreatment(anyprior-awardinformation)versuscontrol(noprior-awardinformation)onaccuracyforpunitivedamagesdata(left)andpainandsufferingdata(right),eachtruncatedatits99thpercentile($50millionand$100million,respectively).Redlinesshowobservedteststatistics.

202

TableC.EffectsonError

Effect

PunitiveDamages(truncatedat99thpercentile($50million))

PainandSuffering(truncatedat99thpercentile($100million))

MSE(trillionsof

dollars)

Differencefromcontrol(MSE=146)(trillionsofdollars)

(95%FisherInterval)

Unadjustedp-value105

MSE(trillionsof

dollars)

Differencefromcontrol(MSE=492)(trillionsofdollars)(95%FisherInterval)

Unadjustedp-value

Treatment(anyprior-awardinformation)

33 -114(-141,-80)

0.000*** 192 -300(-423,-156)

0.008**

Unbiasedprior-awardinformation

56 -103(-132,-70)

0.000*** 221 -271(-404,-122)

0.002***

Unbiasedlowvariabilityprior-awardinformation

19 -127(-161,-91)

0.000*** 230 -262(-415,-101)

0.008**

Unbiasedhighvariabilityprior-awardinformation

57 -89(-122,-52)

0.001** 261 -231(-381,-71)

0.013*

Biasedprior-awardinformation

22 -124(-153,-91)

0.000*** 163 -329(-460,-183)

0.000***

Biasedlowvariabilityprior-awardinformation

24 -123(-157,-86)

0.000*** 131 -361(-515,-195)

0.000***

Biasedhighvariabilityprior-awardinformation

30 -116(-151,-79)

0.000*** 109 -383(-532,-221)

0.001***

105Throughoutthissection,starsindicatestatisticalsignificanceaftercorrectionformultiplecomparisons.Inparticular,***denotesstatisticalsignificanceattheα=0.001level,**denotessignificanceattheα=0.01level,and*denotessignificanceattheα=0.05level.

203

FigureV.2.Observedmeansquarederror(MSE)(intrillions)fordifferenttreatmentcombinationsforpunitivedamages(left)andpainandsuffering(right).DottedlinesshowMSEforcontrolgroup,andstarsindicatestatisticalsignificance(aftercorrectionformultiplecomparisons)ofthedifferenceinMSEbetweeneachtreatmentcombinationandcontrol.

Asdiscussedabove,prior-awardinformationisexpectedtocausethepossible

introductionoferrorintheformofdistortion,orbias,butalsoreduceerrorbyreducingthe

dispersionofawards.Inbothlevelsofscenario,thebeneficialeffectondispersion

dominatesanydistortionaryeffectonthesizeoftheaward,causinganimprovementin

accuracy.Inbothlevelsofscenario,thereisasignificantpositiveeffectonaccuracy

overall—comparingallactivetreatmentconditionscombined(i.e.,anytreatment

combinationthatinvolvesanyprior-awardinformation)tothecontrolconditionofno

prior-awardinformation(p<0.000***forpunitivedamagesandp=0.008**forpainand

suffering).Furthermore,prior-awardinformationimprovesaccuracywhethertheprior-

awardinformationisunbiasedlowvariability(p<0.000***forpunitivedamagesand

p=0.008**forpainandsuffering),unbiasedhighvariability(p=0.001**forpunitivedamages

andp=0.013*forpainandsuffering),biasedlowvariability(p<0.000***forpunitive

204

damagesandp<0.000***forpainandsuffering),orbiasedhighvariability(p<0.000***for

punitivedamagesandp=0.001**forpainandsuffering).106

Moreover,toconfirmthatourresultsholdwithouttheinfluenceofthemore-

extremevalues,weexaminefourimportanteffectsusingthepunitivedamagesandpain

andsufferingdatatruncatedatthe90thpercentile—i.e.,truncatedat$5millionforpunitive

damagesand$10millionforpainandsuffering.Usingthesedata,weneverthelessobserve

asignificantimprovementinaccuracycausedbyunbiasedlowvariability(p<0.000***for

punitivedamagesandp<0.000***forpainandsuffering)andbiasedlowvariability

(p<0.000***forpunitivedamagesandp<0.000***forpainandsuffering)prior-award

informationforbothlevelsofscenario.Ourresultsfortheseeffectsaresummarizedin

TableDandFigureV.3below.

106Weanalyzetheeffectofformonaccuracyaswell.Thedataprovideevidencethatprior-awardinformationgenerallyimprovesaccuracyregardlessofform—i.e.,regardlesswhethersuchinformationispresentedasanaverage,list,orrangeofpriorawards.

205

TableD.EffectsonErrorWhenTruncatingatthe90thPercentile

Effect

PunitiveDamages(truncatedat$5million)

PainandSuffering(truncatedat$10million)

MSE(trillions

ofdollars)

Differencefromcontrol(MSE=4.29)

(trillionsofdollars)

(95%FisherInterval)

Unadjustedp-value

MSE(trillions

ofdollars)

Differencefromcontrol(MSE=16.4)

(trillionsofdollars)(95%FisherInterval)

Unadjustedp-value

Unbiasedlowvariabilityprior-awardinformation

1.39 -2.89(-3.7,-2.1)

0.000*** 11.3 -5.19(-7.5,-3.1)

0.000***

Biasedlowvariabilityprior-awardinformation

1.37 -2.92(-3.8,-2.2)

0.000*** 6.57 -9.9(-12.2,-7.7)

0.000***

206

FigureV.3.Observedmeansquarederror(MSE)(intrillionsofdollars)forunbiasedlowvariabilityandbiasedlowvariabilityconditionsforpunitivedamagesdata(left)andpainandsufferingdata(right),whentruncatedatthe90thpercentileineachscenario.DottedlinesshowMSEforcontrolgroup,andstarsindicatestatisticalsignificance(aftercorrectionformultiplecomparisons)ofthedifferenceinMSEbetweeneachtreatmentcombinationandcontrol.

OurresultsthusprovidestrongempiricalsupportfortheconclusioninTheLogicof

CCG,thatprior-awardinformationimprovesaccuracyundervariousconditionsofbias,

variability,andform.

4.3 EffectofPrior-AwardInformationonSpread

Theeffectsofprior-awardinformationonaccuracyevidenceeffectsonthe

dispersionofawardsaswell.Specifically,becauseaccuracyisdefinedusingMSE,whichcan

beseparatedintobiasandvariance,andbecauseprior-awardinformationcannotreduce

bias(ordistortion),weknowthatanyimprovementinaccuracyisduetoareductionin

variance.However,weseparatelytesttheeffectofprior-awardinformationonspread,

whichwedefinedifferently,usingtheIQRratherthanvariance.Understandingtheeffectof

207

prior-awardinformationonspread,inadditiontoitseffectsonaccuracy,permitsamore

nuancedunderstandingofthedata.

TheIQRis,inasense,lesssensitivetodifferencesinrandomvariation.Forexample,

changingavalueatthe95thpercentilefrom$1millionto$50millionwouldnotaffect

spread.Butspreadprovidesspecificinformation—thedifferencebetweenthe75th

percentileandthe25thpercentiles—thatmaybeobscuredinothermeasuresofrandom

variation.

Thedataprovidestrongevidencethatprior-awardinformationreducesspread.In

thepunitivedamagesscenario,ourcomparisonofallactivetreatmentconditions

(combined)tothecontrolconditionindicatesthat,overall,prior-awardinformationcauses

asignificantreductioninspread(p<0.000***).Furthermore,wefindthatbothunbiased

prior-awardinformationandbiasedprior-awardinformationcauseareductioninspread

(p=0.02*forunbiasedandp<0.000***forbiased).107

Inthepainandsufferingscenario,wedetectasignificantreductioninspreadcaused

bybiasedprior-awardinformation(p<0.000***),butnosignificanteffectcausedby

unbiasedprior-awardinformation.Thislatterresultisnotverysurprising,becausethe

rangeofthepriorawardsinthepainandsufferingscenario—andparticularlythehigh

107Note,wefindthattheaveragelevelofformtendstohaveagreaterdownwardimpactonspreadrelativetotheotherlevelsofform.Thismaybeinterpretedasresultingfromtheparticipants’perceptionthatthereisnovariabilityintheprior-awards,sincetheywereprovidedonlyasinglenumber.Alternatively,theparticipantscouldhaveinterpretedtheaverageformasprovidinglessinformationandtherefore“deservingof”lessinfluence.Participantssimplydidnotknowwhethertheaveragereflected,forexample,fivepriorawardsofidenticalvaluesorfivehighlyscatteredpriorawards.Onbalance,theyseemtohave“interpreted”(likelyimplicitlyratherthanactively)theinformationasreflectingawardsoflowervariability.

208

variabilityprior-awardinformation—isfarlargerthanthecontrolgroupIQR.108This

causesspreadtoremainunchanged(i.e.,withoutsignificanteffect),andeventoincreasein

responsetohighvariabilityprior-awardinformation,notwithstandinganoverallreduction

inrandomvariation,asreflectedintheeffectsofunbiasedprior-awardinformationon

accuracyinthepainandsufferingscenario.109Effectsonspreadwithineachlevelofscenario

aresummarizedinTableEbelow.FisherintervalsareprovidedinTableEforthe

differenceinspreadbetweeneachtreatmentlevelandcontrol,wherethecontrolgroupfor

punitivedamageshasaspreadof$2,900,000andthecontrolgroupforpainandsuffering

hasaspreadof$4,500,000.TableFsummarizessupportfromthedatafortheexplanation

above,andinnote109,reflectingouranalysisregardingtherelationshipbetween1)the

108Thereasonsforthisaretwofold:1)thepercentileschosenfordeterminingtherangeofvaluesofunbiasedhighvariabilitypriorawardsarethe15thand85thpercentiles,substantiallywiderthanthe25thand75thpercentiles;and2)thereisgreaterspreadinthepilotstudydatathaninthecontrolgroupdata,causedby“choppiness”inthedataorsamplingvariationresultingingreaterdispersioninthepilotstudyawards.

109Unbiasedhighvariabilityandunbiasedlowvariabilityprior-awardsinthepainandsufferingscenariorangefrom$200,000to$15millionandfrom$1.12millionto$4.7million,respectively,comparedtothe25thand75thpercentilesofthepainandsufferingcontrolgroupawards,whichare$50,000and$5million,respectively.Thisexplanationiscorroboratedbyourresultsinthepunitivedamagesscenario,where,althoughspreaddecreasesinresponsetounbiasedandbiasedprior-awardinformation(separatelyandcombined),anddecreasesinresponsetounbiasedlowvariability,biasedlowvariability,andbiasedhighvariabilityprior-awardinformation,itincreasesinresponsetounbiasedhighvariabilityprior-awardinformation.Thismakessense,because,aswiththepainandsufferingscenario,unbiasedhighvariabilitypriorawardsrangeinvaluefrom$10,000to$10million,fargreaterthanthecontrolgroup25thand75thpercentiles,$100,000and$3million,respectively.(Comparethistotheunbiasedlowvariabilitypriorawards,whichrangeonlyfrom$500,000to$1million,wellwithintherangeofcontrolgroup25thand75thpercentiles.)Thisexplanationisfurthersupportedbypreviousfindingsthatinformingjurorsofadamagescapmaycauseanupwardeffectonawardamountsanddispersion.SeeSaks,etal.,supranote20,at249-53.Notethatsuchaneffectisnotobservedforbiasedprior-awardinformation,sincetheawardsinthosetreatmentconditionsarefarlessvariable(andhavefarlowerranges)thantheawardsintheunbiasedtreatmentconditions.Wethusobserveasignificantreductioninoverallspreadcausedbybiasedprior-awardinformationinbothlevelsofscenario(evendespitetheabsenceofadetectedreductioninmagnitudeinthepainandsufferingscenario(discussedinfurtherdetailinfra)inresponsetobiasedprior-awardinformation).

209

rangeofthepriorawardsprovidedtoparticipantsrelativetothecontrolgroupIQR,and2)

theeffectoftheprior-awardinformationonspread.

210

TableE.EffectsonSpread

Effect

PunitiveDamages PainandSuffering

Spread(IQR)

Differencefromcontrol(IQR=

2,900,000)(95%FisherInterval)

Unadjustedp-value

Spread(IQR)

Differencefromcontrol(IQR=

4,500,000)(95%FisherInterval)

Unadjustedp-value

Treatment $1,430,000 -$1,470,000(-$1,620,000,-$1,070,000)

0.000*** $4,978,000 $478,000(-$610,000,$2,400,000)

0.603

Unbiasedprior-awardinformation

$2,351,000 -$549,000(-$749,000,-$70,000)

0.018* $6,000,000 $1,500,000(-$72,000,$3,510,000)

0.104

Unbiasedlow

variabilityprior-awardinformation

$427,500 -$2,472,500(-$2,720,000,-$1,980,000)

0.000*** $2,750,000 $-1,750,000(-$3,250,000,$320,000)

0.109

Unbiasedhigh

variabilityprior-awardinformation

$4,825,000 $1,925,000($1,675,000,$2,420,000)

0.000*** $10,750,000 $6,250,000($4,750,000,$8,310,000)

0.000***

Biasedprior-awardinformation

$509,000 -$2,391,000(-$2,580,000,-$1,920,000)

0.000*** $3,955,000 -$545,000(-$1,650,000,$1,430,000)

0.524

Biasedlowvariabilityprior-awardinformation

$172,500 -$2,727,500(-$2,970,000,-$2,280,000)

0.000*** $1,887,500 -$2,612,500(-$4,110,000,-$566,000)

0.017*

Biasedhighvariabilityprior-awardinformation

$900,000 -$2,000,000(-$2,280,000,-$1,490,000)

0.000*** $6,000,000 $1,500,000($0,

$3,560,000)

0.152

211

TableF.RatiosofRangetoIQRandEffectsonSpread PunitiveDamages PainandSuffering

Rangeofawardspresented

25thpercentile–75th

percentileofcontrol

Ratioof

rangeto

IQR110

Effectonspread

Rangeofawardspresented

25thpercentile–

75thpercentileofcontrol

RatioofrangetoIQR

Effectonspread

Unbiasedhigh

variability

$10,000-$10million

$100,000-$3

million344% +

$200,000-$15million

$500,000-$5million 329% +

Unbiasedlow

variability

$500,000-$1million

$100,000-$3

million17% -

$1.12million-$4.7million

$500,000-$5million 80% 0

Biasedhigh

variability

$1,000-$1

million

$100,000-$3

million34% -

$100,000-$7.5million

$500,000-$5million 164% 0

Biasedlow

variability

$50,000-

$100,000

$100,000-$3

million2% -

$560,000-$2.35million

$500,000-$5million 40% -

Toconfirmourinterpretationofourfindingsregardingspread,andtotestits

robustnesstoalternativetruncationthresholdsandmeasuresofdispersion,wetestthe

effectsofunbiasedprior-awardinformationonthestandarddeviationofaward

determinationsusingthedataforeachscenariotruncatedatthe90thpercentile.Thesetests

providesubstantialsupportforourfindingsandinterpretationabove.Specifically,inboth

levelsofscenario,weobservethatunbiasedprior-awardinformationsignificantlyreduces

thestandarddeviationofawardsrelativetocontrol(p<0.000***forpunitivedamagesand

110CalculatedaslengthofrangedividedbylengthofIQR.

212

p<0.000***forpainandsuffering).Furthermore,inbothlevelsofscenario,weobservethat

unbiasedlowvariabilityprior-awardinformationsignificantlyreducesstandarddeviation

relativetocontrol(p<0.000***forpunitivedamagesandp<0.000***forpainandsuffering),

whereasunbiasedhighvariabilitypriorawardinformationhasnosignificanteffecton

standarddeviationrelativetocontrol(asexpected,duetotherelatively-largedispersionof

unbiasedhighvariabilitypriorawards).

Oneimplicationofourresultswithrespecttospreadisthat,althoughcourtsneed

notbeconcernedthatprior-awardinformationwillreduceoveralldispersionand

accuracy,courtsshouldbecognizantofthepossibilitythatprovidingpriorawardsthat

haveveryhighvariabilitymay,undercertainconditions,increasespread—evenwhile

reducingdispersionoverall—andthushaveamitigatingeffectonthebenefitsofprior-

awardinformationforaccuracy.

Significantly,inadditiontotheeffectsonspread,thesetestsconfirmourbelief

regardingtherobustnessoftheeffectofprior-awardinformationonaccuracy.Specifically,

notwithstandinganydetrimentalconsequencesofthesizeabledispersionofthehigh

variabilitypriorawards(andunbiasedhighvariabilitypriorawardsinparticular),we

observethatprior-awardinformationhasasignificantlypositiveeffectonaccuracyacross

theboard.

4.4 EffectofPrior-AwardInformationonMagnitude

Ourinterestisintestingwhetherunbiasedandbiasedprior-awardinformation(as

wellasinteractionswiththevariabilityofpriorawards)causeanydistortion(or“bias”)in

thesizeofawards.PursuanttoourAnalysisProtocol,webeginbyexaminingtheeffecton

213

magnitude,usingthemeanofthelog-transformeddataforeachscenarioandwithinlevels

ofbiasandvariability.Weobserve,however,thatthismeasuremayresultinsomewhat

misleadingresultsforthecurrentstudy.

Themeanofthelog-transformeddataisacommonly-usedmeasurefortesting

magnitudewithright-skeweddata.Thereasonthatitispopularforright-skeweddatais

thatit“pullsin”extremedatapointsmorethanit“pullsin”moderatedatapoints.Under

certainconditions,however,usingthemeanofthelog-transformeddatacandistortthe

signofanobservedeffect.Thiscanoccur,asitdidinthecurrentstudy—and,likely,

becauseparticipantschosevaluesfromtheirimagination—whensomeawardsare

sufficientlyextremethattheobservedeffectofthenon-transformeddataisdominatedby

theeffectoftheextremeawards.Therefore,whenthoseawardsare“pulledin”more

substantiallythanmoremoderateawards,theobservedeffectisreversedorotherwise

distorted.

Now,arguably,themeanofthelog-transformeddataisamoreappropriatemeasure

withwhichtotestmagnitude,sinceitmitigatestheotherwise-overwhelminginfluenceof

outliers.Inthisstudy,however,thisislikelynotthecase.Specifically,althoughweobserve

anincreaseinthemagnitudeofawardsresultingfromunbiasedprior-awardinformation

(forbothpainandsufferingandpunitivedamages),mosttestsinvolvingothermeasuresof

awardsizeresultinnosignificanteffect.Forexample,usingthemeanresultsinno

significanteffectofunbiasedprior-awardinformationonawardsizeinthepainand

sufferingscenario,andanegativeeffectofunbiasedprior-awardinformationonawardsize

inthepunitivedamagesscenario(andnoeffectifthepunitivedamagesdataistruncatedat

214

the99thpercentile).Itispossiblethattheseeffectsareheavilyinfluencedbyextreme

values;but,arguably,theinfluenceofextremevaluesissignificantinthisstudy—

particularlywherethedatasetsaretruncatedatthe99thpercentile,sincethesedatasets

alreadyexcludethemostextremedatapoints.

Furthermore,forbothpunitivedamagesandpainandsuffering,weobservedno

differencesbetweenthemedianofthecontrolgroupawardsandthemedianoftheawards

inconditionsinvolvingunbiasedprior-awardinformation.Asopposedtothemeanofthe

unloggeddata,thismeasureexcludestheeffectofextremevalues.Wealsotestedtheeffect

ofunbiasedprior-awardinformationonawardsizeusingasameasureofsizethemeanof

awardsinthedatatruncatedatthe90thpercentile,findingnosignificanteffectforpunitive

damagesandapositiveeffectforpainandsuffering.Wesummarizetheeffectsofunbiased

prior-awardinformationonawardsizeunderalternativemeasurementsinTableGbelow.

TableG.Unbiasedvs.Control:AlternativeMeasurementsandCorrespondingEffects

onAwardSize PunitiveDamages PainandSuffering

Logmean Median Mean

Mean(truncatedat90th

percentile)

Logmean Median Mean

Mean(truncatedat

90thpercentile)

Unbiasedversuscontrol Positive 0 Negative 0 Positive 0 0 Positive

Similarly,forbiasedprior-awardinformationinthepainandsufferingscenario,

usingeitherthemeanormedianoftheunloggeddatarevealsnosignificanteffecton

215

magnitude,relativetoawardsinthecontrolcondition,whereasusingthemeanofthe

loggeddataindicatesapositiveeffect,relativetoawardsinthecontrolcondition(p=0.02*).

Thus,ourresultsregardingtheeffectofunbiasedprior-awardinformationwith

respecttoawardsizecanbedescribedasinconclusive.Although,usingthemeanofthelog-

transformeddata,wedetectapositiveeffectofunbiasedprior-awardinformationon

magnitude(p<0.000***forpunitivedamagesandp<0.000***forpainandsuffering),atbest,

thetestsareverysensitivetothemeasureused,and,atworst,thelog-transformcauses

dramaticdistortions.Inanyevent,usingothercommonmeasuresofsize,suchasthe

median,wedetectnosignificantdifferencebetweenthesizeofawardsinthecontrol

conditionandthesizeofawardsintheunbiasedconditions.

Ofcourse,wecanattempttointerpretthisbreakdownofeffects.Onewayof

understandingitisasfollows:unbiasedprior-awardinformationhasnosignificanteffect

onthesizeofawards,inthesensethatthecentralawardisnotaffected.Forexample,inthe

punitivedamagesscenario,themedianofawardsinboththecontrolconditionandacross

theunbiasedconditionsis$1million.Ontheotherhand,unbiasedprior-awardinformation

hasnoeffectorasignificantnegativeeffectonawardsizewhentheeffectsofextreme

awardsareconsidered.Thisiscorroboratedbyourexploratoryanalysisregarding

outliers,111andissignificantinthecurrentstudy,sinceextremeawards,aswellasrandom

variationgenerally,istheproblemthatprior-awardinformationseekstoaddress.Finally,

unbiasedprior-awardinformationmaycausegreaterawardsize,inthesensethatthe

111Seeinfra§4.6.

216

awardsintheunbiasedconditionsaregreaterwhentheeffectofmoreextremeawardsis

excludedbuttheeffectofsomelevelofextremenessisincluded.Onepossibleexplanation

forthiseffectisthattheobserveddetrimentaleffectofthesizeabledispersionofhigh

variabilityprior-awardinformationonspreadmayalsocausesomepositiveeffecton

awardsize,duetothelopsidednatureofthehighvariabilitypriorawards,whichare

boundedfrombelowat$0butunboundedfromaboveandrightskewed.Inanyevent,this

lattereffectisinteresting,butnotveryrobust,sincetheresultsreverttoinsignificance

whenweuse(unlogged)datatruncatedatthe99thpercentiletoexcludeextremeoutliers,

andmixedresultswhenweuse(unlogged)datatruncatedatthe90thpercentile.

Additionally,inlinewithourexpectations,forbiasedprior-awardinformationinthe

punitivedamagesscenario,thedataevidenceasignificantnegativeeffectonmagnitude

(measuredusingthemeanoftheloggeddata)(p<0.000***).Thisresultisconsistentwith

theeffectdirectionobservedusingothermeasuresofawardsize,suchasthemedianor

mean.Further,forpunitivedamages,theoveralleffectofprior-awardinformationon

magnitude,relativetoawardsinthecontrolcondition,isnotsignificant.Similarly,we

observenosignificanteffectonawardsizewhenmeasuredusingthemeanofthelogged

dataorthemedian,andadownwardeffect(inlinewithourexpectations,duetotheeffect

ofbiasedprior-awardinformation)whenawardsizeismeasuredusingthemeanoftheraw

data(p<0.000***).

Finally,regardlesswhetherprior-awardinformationaffectsawardsize,inalllevels

ofbiasandvariability,andforbothlevelsofscenario,weobservethatprior-award

informationimprovesaccuracy.

217

4.5 EffectofLowVariabilityPrior-AwardInformationRelativetoHigh

VariabilityPrior-AwardInformation

Wefindstrongevidence,inlinewithourexpectations,thatlowvariabilityprior-

awardinformationismoreinfluentialthanhighvariabilityprior-awardinformation.

First,forbothlevelsofscenario,thedataindicatethatlowvariabilityprior-award

informationcausesareductioninspread(p<0.000***forpunitivedamagesandp<0.000***

forpainandsuffering)andareductioninmagnitude(p<0.000***forpunitivedamagesand

p<0.000***forpainandsuffering),relativetohighvariabilityprior-awardinformation.

PursuanttothemodelinTheLogicofCCG,lowvariabilityprior-awardinformationprovides

moreinformationthanhighvariabilityprior-awardinformation,112andthushasagreater

influenceonawarddeterminations.This“influence”translatestogreaterimpactsonspread

andmagnitude.

Thus,forbothlevelsofscenario,unbiasedlowvariabilityprior-awardinformation,

relativetounbiasedhighvariabilityprior-awardinformation,hasnosignificanteffecton

magnitudeandasignificantnegativeeffectonspread(p<0.000***forpainandsufferingand

p<0.0000.***forpunitivedamages).Theseresultsareinlinewithourexpectations:

unbiasedprior-awardinformationisexpectedtohavenosignificanteffectonmagnitude,

whetheritislowvariabilityorhighvariability,andprior-awardinformationisexpectedto

haveagreaternegativeimpactonspreadifitislowvariabilitythanifitishighvariability.

112Seesupra§2.3.

218

Moreover,forbothlevelsofscenario,biasedlowvariabilityprior-awardinformation,

relativetobiasedhighvariabilityprior-awardinformation,hasasignificantnegativeeffect

onbothmagnitude(p<0.000***forpainandsufferingandp<0.000***forpunitivedamages)

andspread(p<0.000***forpainandsufferingandp=0.003**forpunitivedamages).These

resultsarealsoinlinewithourexpectations.Specifically,becausetheprior-award

informationisbiased,therearetwoeffects—lowvariabilityprior-awardinformationhas

greaterinfluencethanhighvariabilityprior-awardinformation,andthereforebotha

greaterdistortionary,or“biasing,”effectonmagnitude(here,anegativeeffect),anda

greater“narrowing”effectonspread.Ourresultsregardingtheeffectsoflowvariability

prior-awardinformationrelativetohighvariabilityprior-awardinformationare

summarizedinTableHbelow.

TableH.LowVariabilityvs.HighVariabilityPrior-AwardInformation

PunitiveDamages(Raw)PainandSuffering(truncatedat99th

percentile($100million))

OutcomeObservedtest

statistic

Unadjustedp-value

Observedteststatistic

Unadjustedp-value

Lowvariabilityvs.highvariability

MSE(intrillions) 100,692 0.825 -4.50 0.930

Spread -$2,562,000 0.000*** -$6,056,250 0.000***Magnitude -0.38 0.000*** -0.35 0.000***

Lowvariabilityunbiasedvs.highvariabilityunbiased

MSE(intrillions) -4,544 0.827 -31 0.758

Spread -$4,397,500 0.000*** -$8,000,000 0.000***Magnitude -0.09 0.591 -0.18 0.165

Lowvariabilitybiasedvs.highvariabilitybiased

MSE(intrillions) 205,928 0.493 22 0.803

Spread -$727,500 0.004** -$4,112,500 0.000***Magnitude -0.67 0.000*** -0.52 0.000***

219

Note,apossibleargumentagainstthisinterpretationisthattheresultsobservedcan

beexplainedbytheeffectofhighvariabilityonspreadratherthantheeffectoflow

variabilityontheinfluenceoftheprior-awardinformation.Specifically,asdiscussedabove,

itispossiblethatthehighvariabilityprior-awardinformationcausesanobservedincrease

inspread,sincetherangeofthehighvariabilitypriorawardsisgreaterthantheIQR.Itis

thereforepossiblethattheobservedeffecthereresultsfromremovinghighvariability

awardsandthusremovingtheupwardeffectonspread.Indeed,suchaneffectcouldalso

explainanobserveddecreaseinspreadcausedbyunbiasedlowvariabilityprior-award

informationrelativetounbiasedhighvariabilityprior-awardinformation.Inotherwords,it

ispossibletoarguethattheobservedreductioninspreadhasnothingtodowiththe

influenceoflowvariability,versushighvariability,prior-awardinformation,butrather,that

itiscausedmerelybyremovingtheupwardeffectonspreadcausedbythesizeable

dispersionofthehighvariabilitypriorawards.

Itismoredifficult,however,toapplythisexplanationtotheeffectsofbiasedlow

variabilityprior-awardinformation,relativetobiasedhighvariabilityprior-award

information,withrespecttospreadandmagnitude.Incontrasttounbiasedprior-award

information,itisunlikelythateitherbiasedlowvariabilityprior-awardinformationor

biasedhighvariabilityprior-awardinformationwouldcauseanincreaseinspread,relative

tothecontrolcondition.Indeed,inbothlevelsofscenario,neitherbiasedlowvariability

prior-awardinformationnorbiasedhighvariabilityprior-awardinformationhassuchan

effect.Thisisbecausethevariabilityofsuchinformationisreducedsubstantiallybythe

220

biasratio,113leadingtomutedlevelsofvariabilityforlowvariabilityandhighvariability

prior-awardinformationinthebiasedconditionsrelativetotheircounterpartsinthe

unbiasedconditions.

Furthermore,rememberthatbiasedlowvariabilityprior-awardinformationdoes

notinvolvepriorawardsthatareofloweramountthanbiasedhighvariabilityprior-award

information;rather,itonlyhaslowervariability.Thebiasedlowvariabilityprior-award

information(rangingfrom$50,000to$100,000)iscompletelycontainedwithintherange

ofvaluesinthebiasedhighvariabilityprior-awardinformation(rangingfrom$1,000to$1

million).

Thus,theobservedsignificantnegativeeffectsofbiasedlowvariabilityprior-award

information,relativetobiasedhighvariabilityprior-awardinformation,onspreadand

magnitude,andwithinbothlevelsofscenario,providestrongevidencethatlowvariability

prior-awardinformationhasgreaterinfluence—i.e.,ismoreimpactful—onaward

determinationsthanhighvariabilityprior-awardinformation.Thisfindingprovides

supportforthemodelassumptioninTheLogicofCCG,thattheinfluenceofprior-award

informationisproportionaltotheinverseofthevariabilityoftheawards.

4.6 TheEffectofPrior-AwardInformationonOutliers

Inourexploratoryanalysis,weexaminetheeffectofprior-awardinformationon

“outliers,”whichwedefineasawardsatorabovethe99thpercentile($50millioninthe

punitivedamagesscenarioand$100millioninthepainandsufferingscenario).

113Seesupra§3.

221

Inthecontrolgroupofthepunitivedamagesscenario,therearethirteen

observationsoutof249(5.2%)above$50million,whereasinthetentreatmentarmsof

thepunitivedamagesscenario,therearetwenty-threeobservationsoutof2502(0.9%)

above$50million,or,onaverage,approximatelytwoobservations(23/10=2.3)above$50

millionper249observations.Thisrepresentsasignificantreductioninoutliers( χ 21 = 29.2,

p<0.000**).Furthermore,inthefiveunbiasedtreatmentarmsinthepunitivedamages

scenario,therearefourteenobservationsoutof1257(1.1%)above$50million,or,on

average,approximatelythree(14/5=2.8)above$50millionper249observations.This

alsorepresentsasignificantreductioninoutliers( χ 21 =17.6, p<0.000***).

Inthecontrolgroupoftheuntruncateddatasetforthepainandsufferingscenario,

therearetenobservationsoutof244(4.1%)above$100million,whereasintheten

treatmentarmsofthepainandsufferingscenario,thereare33observationsoutof2463

(1.3%)above$100million,or,onaverage,approximatelythreeobservations(33/10=3)

above$100millionper249observations.Thisrepresentsasignificantreductioninoutliers

( χ 21 = 9.11, p=0.002**).Furthermore,inthefiveunbiasedtreatmentarmsinthepainand

sufferingscenario,therearenineteenobservationsoutof1229(1.6%)above$100million,

or,onaverage,approximatelyfour(19/5=3.8)above$100millionper249observations.

Thisalsorepresentsasignificantreductioninoutliers( χ 21 = 5.6, p=0.018*).

Ouranalysisregardingtheeffectofprior-awardinformationonoutliersis

summarizedinTableIbelow.

222

TableI.AbsoluteDifferencesinProportionsofOutliers PunitiveDamages(Raw) PainandSuffering(Raw)

Treatmentrate

Controlrate

Unadjustedp-value

Treatmentrate

Controlrate

Unadjustedp-value

Treatmentvs.

Control

0.9% 5.2% 0.000*** 1.3% 4.1% 0.002**

Unbiasedvs.

Control

1.1% 5.2% 0.000*** 1.6% 4.1% 0.018*

5. Conclusion

The“starkunpredictability”ofawardsforpainandsufferingandpunitivedamages

isarguablyunacceptable.114Wedesire“predictabilityandproportionality,”115but,atthe

sametime,requirejuriestodeterminetheseawardsthroughnothingmorethana

“standardless,unguidedexerciseofdiscretion.”116Awardsforpainandsufferingand

punitivedamagesshouldbeboundtogether,inthesensethatlikecasesresultinlike

outcomes.Butcourtshavebeenunwillingtobindsuchoutcomestogetheractivelybyusing

awardsincomparablecasesasguidanceforawarddeterminations.

TheLogicofCCGaddressedmajorobjectionstotheuseofprior-awardinformation

toguideawarddeterminations,andarguedthatCCGnotonlyreducesthevariabilityof

awards,butimprovesaccuracy—thatis,biasandvariance—generally.Inparticular,the

paperreliedoncertainbehavioralassumptionstoexplainwhyanyintroductionofbias

causedbyCCGwouldbewelloutweighedbyareductioninvariability;andthatsucheffects

114ExonnShippingCo.v.Baker,554U.S.471,499(2008).

115Paynev.Jones,711F.3d85,94(2dCir.2013).

116Jutzi-Johnsonv.UnitedStates,263F.3d753,759(7thCir.2001).

223

wouldbesufficientlypredictablethat,absentextraordinarycircumstances,courts

generallyneednotbeconcernedregardingwhetherornotCCGwillimproveaccuracyina

particularcase.117Itconcludedthat,“[i]nshort,wecanexpectthatareasonablemethodfor

identifyingprior‘comparable’caseswillresultinprior-awardinformationthatislikelyto

improve[accuracy,andthus,]reliability.”118Furthermore,asecondpaperderivedvarious

mathematicalresultsdemonstratingtherobustnessofCCGmethodstoawiderangeof

conditions,includingthoseinvolvingtheidentificationof“misaligned”priorcases.119But

theseresultsrelyonthesamebehavioralassumptionsasthoserelieduponinTheLogicof

CCG.

Thepurposeofthecurrentstudyistoexaminewhetherthebehavioralassumptions

relieduponinthesepapersarejustified—thatis,whethertheyaresupportedbythedata.

Inparticular,weareinterestedintesting,behaviorally,whetherprior-awardinformation

improvesaccuracyunderarobustsetofconditions,andwhetherjurorsbehaveasthe

modelpredictsinresponsetobiasandvariabilityconditions,andprior-awardinformation

generally.

Inthepresentstudy,wefindthatprior-awardinformationimprovesaccuracy,not

onlywhensuchinformationisunbiased,butalsowhenitisbiased,andevenbiasedandlow

variability.Thatis,evenwhenthepriorawardsarenarrowlydistributedarounda

“misaligned”center—onethatissubstantiallylessthanthe“correct”center—thebeneficial

117SeeTheLogicofCCG§4.

118Id.at21.

119SeeBavli&Chen,supranote50.

224

effectsofprior-awardinformationwithrespecttorandomvariation,orjudgment

variability,aregreaterthanthedetrimentaleffectswithrespecttotheexpectedsizeofthe

award.Thus,anyerrorcausedbythedistortionofawardsize,ortheintroductionofbias,is

outweighedbythereductioninerrorcausedbyreducingdispersion.

Indeed,thisisconfirmed,andgivenadditionalcolor,byourotherfindings.First,itis

clearfromourdescriptivestatisticsthatawardsinthepunitivedamagesscenarioandthe

painandsufferingscenarioareextremelyunpredictable.Second,wefindthat,inadditionto

itseffectonrandomvariationgenerally,prior-awardinformationhasasignificant

downwardeffectonspread.Wealsofindthatithasasignificantdownwardeffecton

outliersinparticular.Thus,prior-awardinformationseemstohavebeneficialeffectson

judgmentvariabilityatagenerallevel—itreducesspread,thefrequencyofoutliers,and

variancegenerally.Note,however,thatanumberoffindingsregardingspreadindicatethat,

ifthepriorawardsaresufficientlyvariable,theymaycausesomesubsetofawardsto

increaseinvariation,evenwhiletherandomvariationofawardsmaydecreaseoverall.

Third,althoughwedonotarriveataconclusivefindingregardingtheeffectofunbiased

prior-awardinformationonmagnitude—e.g.,weareunabletoconfirmtheresultofSakset

al.thatunbiasedprior-awardinformationcausesnochangeintheamountsawardedand

thereforenointroductionoferrorintheformofbias120—thedataprovideevidencethat

biasedprior-awardinformationhasasignificantdownwardeffectonmagnitude,as

predicted.Fourth,inlinewiththebehavioralassumptionsinTheLogicofCCG,wefindthat

120Saksetal,supranote20at249-53.

225

lowvariabilityprior-awardinformationhassignificantlygreaterinfluenceonaward

determinationsthanhighvariabilityprior-awardinformation,andthereforesignificantly

greaterimpactonspreadandmagnitude.

Note,however,thattheeffectofprior-awardvariabilityonspreadandmagnitude

mayinteractwiththeeffectdescribedabove,wherebyhighvariabilitypriorawardsmay

increasethespreadofcertainawarddeterminations.Specifically,itispossiblethat,for

highlyvariablepriorawards,twoeffectsmayinteract:1)thehighlyvariablepriorawards

haveareducedinfluenceonawarddeterminations(relativetolessvariablepriorawards),

resultinginamutedeffectonspreadandmagnitude;and2)thepriorawardsmayhavean

upwardimpactonspreadandmagnitude.Itisunclearwhicheffectwoulddominateina

particularapplicationofCCG;butourfindingssuggestthatthelattereffectwould,atmost,

beconcerningwhenthepriorawardsareextremelyvariable—andeventhen,asevidenced

bythedata,thepriorawardsarelikelytoimproveaccuracyonbalance.

Althoughweuncoversubstantialevidenceinsupportofourhypothesesandthe

argumentinTheLogicofCCG,thereareanumberofimportantlimitationstoourstudy.

First,ashighlightedabove,someresultsgiverisetothepossibilityofeffectsthatarenot

describedbyourhypotheses,orthatweexplainaseffectsrelatedtomeasurement.

Althoughtheseissuesaregenerallyunsurprising,andourexplanationsperhaps

uncontroversial,conclusivereasoningmayrequireadditionalexamination.

Second,theexperimentalunitsinthisstudyaremockjurorsratherthanmockjuries,

whereasjuriesaretheunitsatissueinthecontextofreal-worldawarddeterminations.Of

course,juriesarecomposedofjurors,andthebehaviorofjurorsarelikelycloselytiedto

226

thatofjuries;nevertheless,extendingconclusionsfromthepresentstudytojuriesrequires

additionalinferencethatshouldbeconsideredcarefully.Forexample,itislikelythatthe

randomvariationofawardsislessforjuriesthanjurors,wheretheformerdeliberateand

combine—insomesense,compromiseover—heterogeneousviewsregardingan

appropriateaward.121

Third,ourexperimentalunitsaremockjurors,ratherthanreal-worldjurorswho

decideanawardfollowinganactualtrial(ratherthanadescription).Inasense,thisstudy

isasimplified“laboratory”experimentaimedatstudyingjurorbehavior.Amoreideal

(althoughfarcostlier)experimentwouldinvolveaninterventioninreal-worldaward

determinationsusingjuriesratherthanjurors,andreal-worldjuriesinthecontextofreal-

worldtrials.Forexample,thevariabilityofawardsmaybeexaggerated,becausemock

jurorstreatthesituationashypotheticalandignorecertainrealconsequencesofextreme

awards,suchasbankruptcies,jobloss,etc.122Notethatextremeawardsareobservedinthe

realworld,butlikelylessfrequently(andperhapslessdramatically)thaninsurvey

experiments.Additionally,thesummarydescriptionprovidedtoparticipantsineachlevel

ofscenarioislikelytoaffectthevariabilityofawards(bothincontrolandactivetreatment

conditions)relativetoreal-worldtrialsinwhichmultifacetedevidentiarysupportis

providedtosubstantiateargumentsbytheplaintiffandthedefendant,andpresidedover

byajudge.

121SeeDiamondetal.,supranote7,at316(comparingjurorandjurydamageawards);Sanders,supranote13,at494-96(discussingcriticismsbasedonthedisparitybetweenjuror-basedstudiesandjuryawards).

122Ontheotherhand,itispossiblethatsummarydescriptionsarelessemotion-provokingthanreal-worldtrials,andthereforeweighintheoppositedirection.

227

Fourth,weusedMechanicalTurktoadministerthestudy.Althoughwecompared

thepopulationofMechanicalTurkuserstothepopulationofindividualsonU.S.juries,and

concludedthatthesepopulationsaresimilarbasedonthecovariatesexamined,thereare

likelytobesomedifferencesthatmayaffectourresults.123Additionally,althoughwe

attemptedtoexcludeindividualsnotqualifiedforparticipationonaU.S.jury,neitherour

technicalinclusioncriterianorourstatedinclusioncriteriaensuredcompliance,andwedid

notexpecttoexclude100%ofindividualsineligibleforjuryservice.Totheextentthat

certainparticipantsshouldhavebeenexcludedfromthestudybasedontheirineligibility

toserveonaU.S.jury,andtotheextentthattheirresponsestooursurveydifferfromthe

responseswewouldreceivefromthetruepopulationofeligiblejurors,suchparticipation

mayaffectourresults.

Notwithstandingtheselimitations,andothers,ourfindingsprovidestrongevidence

thatCCGimprovesaccuracyunderarobustsetofconditions,andthatthebehavioral

assumptionsandconclusionsinTheLogicofCCGaresupportedbythedata.Insummary,

ourresearchprovidessubstantialsupportfortheargumentthatCCGisaneffectivemethod

forreducingtheunpredictabilityofawardsforpainandsufferingandpunitivedamages,

whilepreservingthediscretionofthetrieroffactandimprovingtheaccuracyofawards

generally.

123SeegenerallyConnorHuff&DustinTingley,“WhoAreThesePeople?”EvaluatingtheDemographicCharacteristicsandPoliticalPreferencesofMTurkSurveyRespondents,2015Research&Politics1(2015).