1

HowtoEngagethePublicontheEthicsandGovernanceofAutonomousWeaponSystems

JasonMillar1andAJungMoon2

Abstract

ThereareactiveinternationaldiscussionstakingplaceontheethicsandgovernanceofAutonomousWeaponSystems(AWS)—robotsthatcankillwithoutdirecthumaninterventionoroversight.Itisimperativethatwecriticallyexaminetheroleandnatureofpublicengagementintendedtoinformdecisionmakers.TheMartensClause,includedintheadditionalprotocolsoftheGenevaConventions,makesexplicitroomforthepublictohaveasayonwhatisdeemedpermissibleinmattersofarmedconflict,especiallywherenewtechnologiesareconcerned.However,manymeasuresofpublicopinion,usingmethodssuchassurveysandpolls,havebeendesignedinsuchawaythatmakesthemsubjecttopotentialbiasingeffects.Forexample,someonlyconsiderspecificapplicationsinsteadofgeneralaspects/featuresuniquetothetechnologyunderconsideration.Inthispaper,wesurveyvariousstudiesthathavebeenconductedtogaugepublicopinionontheuseofmilitarydrones(autonomousandremotelyoperated),includingtherecentinternationalpollconductedbytheOpenRoboethicsinitiative(ORi).Bydrawingonevidencefrommoralpsychology,wehighlightsomepotentialbiasingeffectsthatparticularquestionframingscouldhaveonoutcomes,andoutlineconsiderationsthatcanbetakenintoaccountwhendesigninganddeterminingtheapplicabilityofpublicopinionmeasurestoquestionsofthegovernanceofAWS.SuchconsiderationscanhelppublicengagementobjectivesliveuptothespiritoftheMartensClause.

Introduction

Thelistofmilitariesusingroboticweaponsystemstoaidincarryingoutcombatoperationsisgrowing(Tucker2014).Today’sroboticweaponsystemsareremotelyoperated,andsometimesreferredtoasRemotelyOperatedWeaponsystems(ROWS).WithROWS,thedecisiontouselethalforce(i.e.tofireaweaponatatarget)remainsahumandecision.However,systemsarebeingdevelopedthatcouldallowmilitaryrobotstoautonomouslymakethedecisiontouselethalforce.AsdefinedbytheUSDepartmentofDefense,thesearesystemsthat,“onceactivated,canselectandengagetargetswithoutfurtherinterventionbyahumanoperator”

1JasonMillarisaPostdoctoralFellowattheUniversityofOttawaFacultyofLaw,andteachesrobotethicsandphilosophyatCarletonUniversity(Ottawa).HeistheChiefEthicsAnalystattheOpenRoboethicsinitiative(openroboethics.org),andtheWorkingGroupsCoordinatorattheFoundationforResponsibleRobotics(responsiblerobotics.org).2AJungMoonisaPhDcandidateattheUniversityofBritishColumbia(Vancouver),andco-founderoftheOpenRoboethicsinitiative(openroboethics.org).Shehasauthoredandco-authoredseveralpapersonRoboethicsandhuman-robotinteraction.

2

(DepartmentofDefense2012[inHorowitz2016a]).Inotherwords,inthenot-too-distantfuturemilitaryrobotscouldkillwithouthumanintervention.

TheprospectofthosenewAutonomousWeaponsystems(AWS)hassparkedanethicaldebatethathasbeentakenupbythepublicmedia(Evelith2014;Parkon2015;Devlin2016),governments(Linetal.2008),civilsociety(PAX2014;HumanRightsWatch2015;Article362015),andtheUnitedNations(UNIDIR2014).OnonesideofthedebatewehavevariousargumentstotheeffectthatalthoughROWSmaybeethicallypermissible,AWSarenot,soAWSshouldbeinternationallybanned(e.g.HumanRightsWatch2015;Article362015).OntheothersideofthedebatewehavevariousargumentstotheeffectthatbothROWSandAWSareethicallypermissibletechnologies,andthatabanisunwarranted(e.g.Arkin2015).

WhatmighthelpinformthedebateovertheethicalpermissibilityofROWSandAWS?

Inthispaper,wefocusonpublicengagementasameansofinformingthedebateovertheethicalpermissibilityofROWSandAWS.Wedonotaimtodirectlyaddressthecentralquestioninthedebate—AreAWSethicallypermissible?Rather,supplementingpreviousworkingeneralsurveydesignliterature3,weaimtoinformthewayresearchersgoaboutengagingthepubliconthat,andrelated,questionsaboutnewweaponstechnologies.Westartbymakingthecasethat,onthisparticularissue,publicengagementisanimportantandnecessarypolicyinput.Wethenidentifyadistinctionbetweentwogeneralkindsofquestionstypicallyusedinthosesurveys:questionsthatfocusontheapplicationofthetechnology,andquestionsthatfocusonthenatureofthetechnology.IfresearchersareinterestedingaugingtheethicalpermissibilityofparticularapplicationsofROWSandAWS,askingapplication-specificquestionswillbeuseful.However,wearguethatapplication-specificquestionscanintroduceproblematicbiasesinsurveysmeanttoshedlightonthedebateovertheethicalpermissibilityofAWS.Wethenexaminetwomoralpsychologicalbiasingeffectsthatareofparticularconcernwhenaskingpeopleaboutrobots:moralemotionalpriming,andanthropomorphicframing.Finally,wesurveyexistingpublicopinionpollsandexaminetheminlightoftheconsiderationsjustoutlined.Whyengagethepublic?Decisionswhetherornottobuildorusenewkindsofweaponsareamatterofdemocraticandhumanitarianconcern.The20thcenturywasparticularlyillustrativeonthismatter.Overthecourseofmeredecades,soldiersandcivilianswereintroducedtonewanduniqueformsofsufferingresultingfrommilitarydecisionstodevelopandusesuchtechnologiesaschemicalweapons,landmines,incendiaryweapons,clustermunitionsandnuclearweaponsonthebattlefield.Variousinternationalgovernanceinitiativeshavearisenovertheyearsinresponsetothenatureofthoseweapons.

3Forabrief,yetexcellent,overviewofquestionnairedesignseethesectiononQuestionnaireDesigninPewResearchCenter(2016).

3

MostnotablearetheGenevaConventions,draftedin1949followingtheSecondWorldWar,whichsetouttherulesofarmedconflictand“remainthecornerstoneofcontemporaryinternationalhumanitarianlaw”(ICRC2009).4TheGenevaConventionsprovidespecificprotectionsforprisonersofwar,non-combatants,civilians,andvictimsofarmedconflict,andhavebeenratifiedby196countries.Theyalsomakeclearthatthosewhoarepartytoarmedconflicthavelimitedrightsinchoosingthemethodsandmeansofwarfare(InternationalJusticeResourceCentre2016).Morespecifically,Article36ofAdditionalProtocolItotheGenevaConventionsexplicitlyraisestheintroductionofnewweaponstechnologiesasaconcern:

“Inthestudy,development,acquisitionoradoptionofanewweapon,meansormethodofwarfare,aHighContractingPartyisunderanobligationtodeterminewhetheritsemploymentwould,insomeorallcircumstances,beprohibitedbythisProtocolorbyanyotherruleofinternationallawapplicabletotheHighContractingParty.”

However,itisnotalwaysclearhowwell,orif,existingrulesapplytonewweapons.Incasesofuncertainty,internationalhumanitarianlaw(IHL)underscorestheimportanceofpublicengagementasameansofinformingdeterminationsofthepermissibilityofnewweaponsystems.TheMartensClause,includedinthePreambleofAdditionalProtocolIIoftheGenevaConventions,makesexplicitroomforthepublictohaveasayonwhatis,andisnot,deemedpermissibleinmattersofarmedconflict,especiallywherenewtechnologiesareconcerned.Itreads:

“TheHighContractingParties,…Recallingthat,incasesnotcoveredbythelawinforce,thehumanpersonremainsundertheprotectionoftheprinciplesofhumanityandthedictatesofthepublicconscience,...”(AdditionalProtocolIItotheGenevaConventions)

ThoughlegalscholarsoftendisagreeonhowbesttointerpretandimplementtheMartensClause(Cassese2000),itremainsafactthat,fromtheperspectiveoftheClause,“thedictatesofthepublicconscience”aremeant,incertaincases,tohelprefineandextendtheinternationallawsofarmedconflict.Indeed,attwoseparatemeetingsoftheUNConventiononCertainConventionalWeapons(CCW),firstin2014andagainin2015,expertsandstaterepresentativesconvenedspecificallytodiscusstheprospectofAWSandconsideraninternationalresponse.TheMartensClausewascitedasarelevantguidinggovernanceconsiderationinthemeetingproceedings.GroupssuchasHumanRightsWatch(2015)andtheICRC(2015)havequestionedwhetherornotcurrentlawsarecapableofaddressingtheuniqueissuesraisedbyAWS.ThesequestionshelptounderscoreakeydistinctionbetweenAWSandROWS:AWStransferthepowertomakelethaldecisionsfromahumantoamachine.Thatdistinctionisattheheart4TheGenevaConventionsof1949amendedthreeexistingconventionsdatingfrom1864,1906and1929andaddedafourth.TheGenevaConventionshavesincebeen“supplementedanddeveloped”bythreeAdditionalProtocols(ICRC2009).

4

ofthecontroversysurroundingAWS.DiscussionsattheUNCCWonAWShavefocusedontheimportanceofmaintaining“meaningfulhumancontrol”overdecisionstouselethalforce.Butitisdifficulttodefinemeaningfulhumancontrol(Article362015;Horowitz&Scharre2015).Forexample,meaningfulhumancontrolcouldbeinterpretedasadesignrequirementjusttomaketheweaponsystemmeettherequirementsofexistingIHL,oritcouldformanadditionalrequirementforaweaponsystemevenifthatweaponsystemcouldconformtoIHL(Horowitz&Scharre2015).Furthermore,theprocessofautomatingdecisionstofireatatargetinvolvesprogressivelyautomatinganumberofdifferentfunctionsovertime.Itmightnotbeentirelyclearwhenaparticularautomationtechnologyhasbecomesoautomatedthatithas“crossedtheline”frommeaningfulhumancontrol,intoanethicallyimpermissibleterritory(Millar2015).TheMartensClauseseemsparticularlyusefulhere,since,ifmeaningfulhumancontrolisanimportantguidingconceptintheethicsofAWS,andiftheMartensClauseappliestotheAWSdebate(aswehaveargueditdoes),thenmeaningfulhumancontrolisboundtothepublicconscience.Thus,wearguethatthechallengingtaskofdefiningmeaningfulhumancontrolcan,andshould,beinformedby,andalignedwith,thepublicconscience.WeassertthattheprospectofAWSisacasethatfitsthepurposeoftheMartensClause.PublicengagementisonewaytosupporttherequirementssetoutintheClause,andtogaintractiononthepermissibilityofAWS.TwokindsofquestionsabouttechnologyPublicopinionpollingcanhelpachievethetasksetoutintheMartensClausebyhelpingustobetterunderstandthepublicconscienceonthepermissibilityofnewweaponstechnologies.However,aswithanysurvey,thequestionsmustbedesignedtoprobeparticipants’opinionontheheartofthematter.Here,ratherthanreiteratingbestpracticesfromsurveydesignliterature,wehighlightsomeofthepitfallsspecifictothediscussionofROWS/AWSissues.IntheROWS/AWSdebate,weassertthatquestionsthatcontainreferencetospecificapplicationscanmisleadparticipants.Rather,questionsmustbedesignedtorevealchangestothepublic’sresponsewithrespecttothenatureofthetechnologiesinquestion.Considerthefollowingtwoquestions:

(A) Shouldyourgovernmentdevelopanduseautonomousweaponsystems?(B) Shouldyourgovernmentdevelopanduseautonomousweaponsystemstoguard

againstterrorism?Question(A)isaquestionaboutthenatureofAWS;itisanapplication-neutralquestion.Inordertoansweritinawaythatwouldsatisfytheresearcher’sintent,onewouldneedtohavespecificknowledgeaboutAWS,perhapsaboutspecificfeaturesuniquetothetechnology,orabouthowthoseuniquefeaturescompareto,say,uniquefeaturesofothertypesofweapons.Assuch,asurveyposingonlyquestion(A)toparticipantsmightrequireapreamblecontaininga(somewhat)substantivedescriptionofAWS,oritwouldruntheriskofconfusingparticipantsandgeneratingrelativelyuninformativedata.Withoutapreambletoinformparticipantsabout

5

thenatureofAWS,questionslike(A)couldbesaidtooccur“inavacuum”(Horowitz2015,3),biasingresultsbycausingparticipantstorecallpopularimagesofso-calledkillerrobots,suchastheTerminator,aneffectwecallTerminatorBias5,whenansweringsurveyquestions.Question(B)isanapplication-specificquestion:itcanbemeaningfullyansweredwithoutanyparticularknowledgeofAWS.ItisaquestionaboutaparticularapplicationforwhichAWScouldpotentiallybeused.Incontrasttoquestion(A),asurveyposingonlyquestion(B)couldrelyonlyontheassumptionthatparticipantshaveanopiniononguardingagainstterrorism.Thisisbecausetheapplicationcoulddotheheavyliftingincaseswhereparticipantsknownothingatallaboutnatureoftheunderlyingtechnology.Toillustratethispoint,considerthefollowingquestion:

(C) Shouldyourgovernmentdevelopandusephasedcyclotronstoguardagainstterrorism?

Dependingonhowcommittedoneistoguardingagainstterrorism,theuseofphasedcyclotronscanberenderedirrelevantinansweringthequestion.Indeed,knowingnothingaboutphasedcyclotronsmighteffectivelyincreaseaparticipant’sfocusontheapplication,exacerbatingtheproblem.Askingquestions(A)and(B)inasurvey,inordertocreatecontext(Horowitz2015)andavoidTerminatorBias,couldbackfire.Thisissofirst,becauseitisnotclearwhether(B)createsmeaningfulcontextoraproblematic,application-specificdistractionintheparticipantand,second,becausetheuseof(B)tocreatecontextcouldleadresearchersawayfromprovidingameaningfuldescriptionofthenatureofatechnology.Nowconsiderafourthquestion:

(D) Shouldyourgovernmentdevelopanduseautonomousweaponsystemstokillknownterrorists’familymembers?

(D)focusesparticipantsonanapplicationthatisbothillegaland(weassume)broadlyconsideredethicallyimpermissible,renderingitsubjecttosocialdesirabilitybias6.Asurveyposingonlyquestions(B)and(D)illustratesafurtherproblemwithapplication-specificquestions.IfthegoalistoinformapolicydebateaboutthepermissibilityofAWS,differencesinanswersto(B)and(D)willhelptoinformdebatesaboutparticularAWSapplications,butmightreveallittleaboutparticipants’viewsonthepermissibilityofAWS.

5MediareferencestoTheTerminatoraresoprevalentinreportingonthisissuethatwefeelit’sworthcoiningthephrase.See,forexample,(seeGibbs2015;Brant2016;McDonald2016,andcountlessmore).6Socialdesirabilitybiasreferstopeople’stendencytoanswersurveyquestionsPaulhus,D.L.(1991).

6

Pairingapplication-specificquestionsfeaturingdifferenttechnologiesmayseemlikeausefulstrategytoisolateavariable,butdoesnotofferasatisfactorysolution.Thispointisillustratedbyconsideringasurveycontainingonlyquestions(B)and(C).Aswithstand-alonequestionsfocusedonaparticularapplication,dependingonasurveyparticipant’sreactiontotheapplication—inthiscaseguardingagainstterrorism—thesurveyresultsmightnotrevealanythinginformativeaboutthedevelopmentanduseofphasedcyclotronsorAWS.Forexample,participantsmightrecognizethatdifferenttechnologiesarefeaturedin(B)and(C)yetanswerbothquestionsthesamewaybecauseofthequestions’heavyrelianceontheapplicationcontext,ortheymightfeeltheneedtoresponddifferentlyjustbecausetheyaretwodifferentquestions.Notallapplication-specificquestionsdistractfromthenatureofatechnology.Applicationsthatareunique,orparadigmatic,ofaparticulartechnologycanhelptodistinguishthosetechnologiesfromothertechnologies.Forexample,AWScanbeusedtofindandkilltargetswithoutdirecthumanintervention.Focusingaquestiononanapplicationthatisuniquetoaparticulartechnologycouldhelptouncoveropinionsonthedistinguishingcharacteristicsofthattechnology.Aswehaveargued,focusingonapplicationsthatarecommontoanumberoftechnologiesormethodsofwagingwar,forexample,guardingagainstterrorism,orkeepingtroopsoutofharm’sway,makedrawingconclusionsaboutthepermissibilityofAWSdifficult.Thus,weshoulddistinguishbetweentechnology-specificapplications(i.e.thoseapplicationsthatareuniquetoaparticularclassoftechnologies),andtechnology-neutralapplications(i.e.thoseapplicationsthatmanydifferentclassesoftechnologiescouldbeputto)whenthinkingaboutstudydesign.MoralpsychologicalbiasingeffectsDifferentstudydesignscantriggerdifferent,complex,psychologicalresponsesinparticipants.Inthissection,wefocusontwopossiblemoralpsychologicalbiasingeffectsthatseemparticularlyrelevantinthecontextofstudiesonROWSandAWS:inducingmoralemotionalresponsesandanthropomorphicframings.Wereviewsomeofthemoralpsychologyliteraturetosetupconsiderationsthatshouldbetakenintoaccountwhenpollingpublicsonthedevelopmentanduseofnewweapons.Anybiasintroducedbymoralemotionalresponses,asopposedtoemotionalresponsesgenerally,couldbeparticularlytroublesomeinsurveysintendedtogaugethe“publicconscience”,preciselybecausethepublicconscienceisameasureofmoraljudgment.Measuringthepublicconscienceisallaboutelicitingmoraljudgments:areROWSorAWSmorallypermissible?However,thelinkbetweenmoralpsychologyandpublicpollinginthecontextofweaponsystemsisrelativelynewand,understandably,underdeveloped.Ourgoalinthissectionismerelytodescribesomemoralpsychologicaleffectsthatwehavereasontobelievecouldintroducebiasinpollingresultswiththeunderstandingthatmeasuringtheeffects,and

7

identifyingworkablesolutionstocounteract,ormitigate,suchbiases,wouldrequiremoreresearch.MoralEmotionalPrimingAngerisamoralemotion.Itisassociatedwith,andoftenaccompaniesjudgmentsofrightandwrong.Othermoralemotionsincludeshame,disgustandguilt(Haidt2013,Prinz&Nichols2010).Perhapsnotsurprisingly,experiencingthesemoralemotionstendstoaffectthequalityofourmoraljudgments.Whenweexperienceanger,forexample,thecharacteristicresponseistoseekrevengeorretribution(Prinz&Nichols2010).Thepsychologicalprofileofangerissomethinglikethis:Amywitnessesanautonomyviolation(i.e.aharmorinjustice)thatresultsinanattributionofblame.Forexample,whenwatchinganewsstoryaboutBobharmingChris,AmyblamesBobfortheharm.AmythenfeelsangryatBobforharmingChris.TheresultisthatAmywantstoseeBobpunishedforharmingChris(Prinz&Nichols2010).AstudyconductedbyLerneretal.(1998)illustratesthiseffect.Researchersinducedangerinonegroupofsubjectsbyshowingthemavideoclipinwhichabully“humiliatesandbeatsupateenager”(Lerneretal.1998,566).Anothergroupofsubjectswasshownavideodepictingabstractfigures,knowntoelicitnoemotionalresponse.Subjectsinbothgroupswerethenshownanothervideoinwhichaperson’snegligenceledtoaninjury.Subjectswereaskedhowharshlythenegligentpersonshouldbepunished.Subjectswhowereprimedintheangerresponsegroupweremorepunitiveintheirjudgmentsthanthoseintheemotion-neutralgroup.Similarangerresponsescanbeevokedmerelybyaskingpeopletorecallaneventinwhichtheyweretreatedunjustly(Mikula1986).Moralemotionalprimingeffectscanresultwhenmoralemotionsotherthanangerareelicited.Inanotherstudythatuseddisgustasthemoralemotionalvariable,twogroupsofparticipantswereshownvideosdepictingmoraltransgressions(e.g.harms),andwerethenaskedtojudgethetransgressionstheyhadjustwitnessed.Participantssittingatadirtytablewhilewatchingthevideotapesjudgedthetransgressionsmoreharshlythanparticipantssittingatcleantables(Schnalletal.2008).Thus,elicitingmoralemotionalresponsesinadesignsetupcanhaveanoticeablebiasingeffectonparticipants’moraljudgments.Theeffectisoftensubtleandintuitive,asdemonstratedintheabovestudies,thoughstatisticallysignificant.Thatis,theparticipantneednotbeawareoftheemotionalresponseforittohaveaneffect.Moralpsychologicalresearchsuggests,therefore,thatwemustbeawareofmoralemotionalpriminginstudydesign.Application-specificquestionsthatprimeparticipantstothinkaboutthingsthatmakethemangryordisgustedcouldbiasanswersbyinducingthecorresponding

8

emotionalresponseintheparticipant:inthecaseofanger,theparticipantcouldformadesireforrevengeorretribution(Prinz&Nichols2010).ThisrelatestoacommonbiasthatmaybeatworkinpollingpeopleaboutROWSandAWS.Forexample,aquestionmentioningterrorismorspecificpeopleconsidered“enemies”,suchasquestion(B)above,couldinduceanangerordisgustresponse.Theeffectscouldbiasanswerstowarddevelopingandusingthoseweaponsasameansofretribution,regardlessofthekindofweaponinquestion.Thiscouldalsobetrueofquestionsfocusedonsavingsolders’lives,orprotectingthemfromthreats.Participantswhoexperienceanangerresponsetothethoughtof“friendly”soldiersdyingatthehandsofenemysoldiersorforeignnationalscouldbesimilarlybiasedtowardsdevelopingandusingwhateverweaponmightpreventwhattheymightperceiveasan“unjust”stateofaffairs.AnthropomorphicFramingsWehaveknownforsometimenowthathumanshaveastrongtendencytoanthropomorphizerobots(Johansson1973;Weizenbaum1976;Duffy2003;Bartnecketal.2007;Kahnetal.2011;2012;Darling2015).Thatmeanswetendtoattributehuman-likepropertiestonon-humanentitiesintheworld.We’llsay,forexample,thatwhenthelittlecolourpinwheelonourcomputerscreenisspinning,ourcomputeris“thinking”,orthatwhentwoobjectsmovecloseorfolloweachother’smotionthattheyareattractedtooneanother,orthatwhenacomputer-generatedopponentinavideogameshootsatus“heistryingtokillme”.Strictlyspeaking,thecomputerisnotthinking,nordoesthecomputer-generatedopponent“want”anythingatall.Yet,thispsychologicaltendencyisnowwelldemonstrated.Kahnetal.(2012)invitedparticipantstointeractwithasocialrobotnamedRobovie,inordertodeterminetowhatextentpeoplewouldanthropomorphizeandattributemoralblametoRobovie.Participants’interactionswithRoboviestartedwithalongconversation,duringwhichRobovieaskedquestions,paidcomplimentsandattemptedjokes.ThislonginteractionwasintendedtoframeRobovieasasocialentity—itwasananthropomorphicframinginwhichRobovieswaspresentedashavinghuman-likesocialabilities.Ultimately,eachinteractionledtoasituationinwhichRoboviemistakenlydeniedeachparticipanta$20“prize”thattheywerepromised.ResearchersdebriefedparticipantsinordertoassesstheiranthropomorphicresponsestoRobovie.AllparticipantsinteractedsociallywithRobovie(e.g.answeredquestions,respondingtorequests,etc.),andthoughtheyheldRobovielessresponsiblethanhumansforbeingdeniedtheprize,participantsheldRoboviemoreaccountablethanavendingmachine(Kahnetal.2012).WhenaskedifRoboviewasconscious,50%answered“yes”,73%indicatedtheybelievedRoboviecouldthink,and63%thoughtRoboviecouldbetrusted(Kahnetal.2012,37).

9

Inanotherexperimentinvolvingasocialrobot,Bartnecketal.(2007)teamedeachparticipantupwitharobottoplayagameofMastermind.7Theresearchersvariedtherobot’sperceivedlevelofintelligencebyvaryingthequalityofitsgameplaysuggestions:the“smart”versionoftherobotmadegoodsuggestionstoitshumanpartner(theparticipant),whilethe“stupid”versionmadeunsuccessfulsuggestions(Bartnecketal.2007,218).Theresearchersalsovariedtherobot’sagreeableness,thatis,the“agreeable”versionoftherobotwaspatientandpolitetoitshumanpartner(theparticipant),whilethe“nonagreeable”versioninsistedthatitwasitsturn.AfterthegameofMastermind,theresearchersaskedtheparticipanttoturntherobotoff,atwhichpointtherobotbeganbeggingtobelefton.Participantspairedwiththe“agreeable”and“smart”robothesitatedsignificantlylongerwhenaskedtoturntherobotoffcomparedtothoseparticipantspairedwiththe“stupid”and“nonagreeable”robot.Bartnecketal.(2007,221)concludethatintelligentrobotsare“perceivedtobemorealive”.Darling(2015)conductedasimilarstudy.Thestudyinvolvedthreeseparategroupsofparticipants.ForeachgroupDarlingaskedparticipantstoobserveasmallbug-likerobotmovingabout,thencrushitwithamallet,andshemeasuredtheirhesitationpriortodoingso.Eachgroupwasintroducedtothelittlerobotswithadifferentframingnarrative.Onegroupofparticipants,thenon-framingcondition,receivednoframing.However,theothertwogroupswerebothsubjecttoanthropomorphicframingduringtheirintroductiontotherobots,thatis,theyweretoldastoryaboutthelittlerobots.ForoneofthosegroupsofparticipantsDarlingframedtheintroductionusinga“personifiedbackstory”inwhichtherobothasaname,“livesinthelab”,“likestoplay”,andsoon.WiththeothergroupDarlingframedtheintroductiontotherobotusinga“non-personified”framing,butonethat“lentitselftoanthropomorphicprojection”,inwhichtherobot“getsaroundbutdoesn’tgotoofar.Lastweek,though,itgotoutofthebuilding”.Participantsintheanthropomorphicframingconditionhesitatedpriortostrikingtherobotssignificantlylongerthanthoseinthenon-framingcondition.AccordingtoDarling,thelongerthehesitation,themoreemotionallyengagedtheparticipant.Darling(2015)concludesthatsubjectingparticipantstoanthropomorphicframingconditionsincreasestheirempatheticresponsetorobots.ThesestudieshavedirectimplicationsforthedesignofstudiesinvolvingROWSandAWS.Questionsthatcastrobotsinsocialroles,suchasguardians,protectors,ethicalagentsordecision-makers,couldinadvertentlyinduceempathetic,orotheranthropomorphic,responsesinparticipants.Forexample,(B),whichmentionsguardingagainstterrorism,couldhaveananthropomorphicframingeffectbycastingtherobotasguardian,andcouldthereforecauseparticipantstofeelsympathytowardtherobot,thusbiasingtheirresponses.Thiseffectcouldalsobetriggeredinasurveycontainingquestionsbothaboutdeployingsoldierstoaccomplishatask,anddeployingrobotstoaccomplishsimilartasks.Questionsaboutsoldierscouldinadvertentlyhaveanthropomorphicframingeffectsthatspilloverintoparticipants’reactionstoquestionsaboutrobots—participantsmightimaginerobotsassoldiersinsuchcases.

7Mastermindisagamethatinvolvesonplayertryingtoguessthecorrectsequenceofcoloured“pegs”thathasbeenarrangedbytheopponent(seehttps://en.wikipedia.org/wiki/Mastermind_(board_game)).

10

Empiricalevidencefrommoralpsychologyandhuman-robotinteraction,likethatmentionedhere,isrelevanttostudydesigninvolvingmilitaryrobots.However,moreworkisrequiredtoconfirmwhetherornotbiasingeffectsaresignificantorproblematicinpractice,howbiasesareintroducedintopollresponses,andhowwemightcontrolforproblematicbiasingeffects.ExistingsurveysonROWSandAWS8Overall,fewpublicopinionpollshavebeenconductedtogaugesupportforROWSorAWS.Inthissection,weexamineexistingsurveysthroughthelensoftheabovediscussiononapplication-specificandapplication-neutralquestions,andmoralpsychologicalbiasingeffects.Ourgoalhereistosurveythekindsofquestionsthathavebeenusedinpublicengagementpolls,andcommentonpotentialbiasingeffectsthatmightbepresentasaresultofstudydesign.WashingtonPost-ABCNewsPoll(2012):Inalargesurveyconductedin2012,theWashingtonPostandABCNews(WP-ABC)posedtwoquestionsrelatedtotheuseof“drones”(inthiscaseROWS)forkillingforeignterroristsuspects,andAmericanterroristsuspectlivingabroad.Theyaskedparticipantstoindicatetheirlevelofapprovalof“Theuseofunmanned‘drone’aircraftagainstterroristsuspectsoverseas.”Inafollowupquestiondirectedatparticipantswhoapprovedoftheuseofdronesforsuchstrikes,theyasked“WhatifthosesuspectedterroristsareAmericancitizenslivinginothercountries?Inthatcasedoyouapproveordisapproveoftheuseofdrones?”Bothofthesequestionsareapplication-specificandtheapplicationsmentionedinthemaretechnology-neutral.Assuch,thesequestionsmightnotprovideanyusefulinformationabouttheuseofdronesspecifically,sinceonecouldanswerthequestionbyrelyingonlyonanopinionaboutkillingforeignterroristsuspects—thefactthatthequestionmentionsdronesastheweaponofchoicecouldfadeintothebackground.Similarly,onecouldanswerthefollow-upquestionbasedonlyonanopinionaboutkillingAmericancitizenswhoarealsoterroristsuspectslivingoverseas,regardlessoftheweaponofchoice.TheWashingtonPost–ABCquestionsalsoseemtoprimeparticipantsalongmoralemotionallines.Bothquestionsmention“terrorists”,whichseemstoruntheriskofinducingangerordisgustresponses.Whetherornotthesebiasingeffectsactuallyplayedasignificantroleintheresultsisanopenquestion.Whatisclear,however,isthattheWashingtonPostinterpretedtheresultsasrevealingsomethingspecificallyaboutAmericans’opiniononthenatureofdronetechnology:

8Note:WhenreviewingeachsurveywefocusonlyonthosequestionsandaspectofthesurveysthatcontainedreferencetoROWSorAWS,orweredirectlyrelated,andignoredemographicquestionsorotherquestionsnotdirectlyaddressingROWSandAWS.

11

thetitleofthereportwas,“TheAmericanPublicLovesDrones”(Cillizza2013).Wethinktheirconclusionisamisinterpretationofthedata.Thearticlemightmoreaccuratelyhavebeentitled,“TheAmericanPublicLovesEliminatingTerroristSuspectsLivingAbroad.”PublicMindAspartofaPublicMind(2013)survey,WoolleyandJenkinsaskedparticipantsfourquestionsintendedtogaugetheiropiniononusing“drones”(inthiscaseROWS)toattackforeignnationalsandAmericancitizenslivingabroad,whohavebeendeemedathreattotheUS.Thesurveyisthereforeapplication-specific,asitisaboutaveryspecificuseofdronesthatcouldbeaccomplishedbyothermeans.Herearethequestions:

1. YouprobablyknowthattheUnitedStatesusesremotelycontrolledaircraftcalled“drones”tospyonandsometimesattackpeopleandotherkindsoftargetsaroundtheworld.Howmuchhaveyouheardorreadaboutthesedrones?

2. Tothebestofyourknowledge,cantheU.S.targetU.S.citizenslivinginothercountrieswithdrones,oristhatillegal?

3. Ingeneral,doyouapproveordisapproveoftheU.S.MilitaryusingdronestocarryoutattacksabroadonpeopleandothertargetsdeemedathreattotheU.S.?

4. Ingeneral,doyouapproveordisapproveoftheC.I.A.usingdronestocarryoutattacksabroadonpeopleandothertargetsdeemedathreattotheU.S.?

Questions(2)-(4)providethemostinformationaboutparticipants’opinionsontheuseofdronesforattackingindividuals.Thoughquestions(3)and(4)areapplication-specific,theyseemtobedesignedtominimizeanymoralemotionalpriming.“PeopleandothertargetsdeemedathreattotheUS”isarelativelyemotionallyneutrallanguage,comparedtotheWashingtonPostsurvey’suseof“terroristsuspects”.Thiscouldhavetheeffectofcontrollingforbiasinducedbythenatureoftheapplication.Itcouldalsohavetheeffectofunderscoringtherolethatdronesareplayinginthesurvey,thatis,itcouldhelpparticipantsfocusontheuseofdronesasthemeanstocarryouttheapplication.Atthesametime,questions(2)-(4)couldpotentiallybeansweredwithoutmuchknowledgeofdrones,owingtotheirapplication-specific,andtechnology-neutral,nature.Question(1)assessesparticipants’knowledgeofthetechnology,andcouldalsohavetheeffectofhelpingresearchersunderstandhowemotionallyprimedtheparticipantsare,thuscontrollingforbias.Forexample,ifaparticipanthasreadorheardalotaboutdrones,andmostpopularmediaaccountsofdronesinvolveemotionallyengaginglanguage,researchersmightassumesomethingaboutthatparticipant’semotionalreactiontodronesingeneral.However,abetterwayofcontrollingforbiascouldbetomeasuretheirlevelofemotionalengagementwhilethinkingaboutdrones.Ofcourse,thiswouldlikelycomplicatetheexperimentalset-up,makingitdifficultorcostlytocarryout.PewResearchCenter

12

In2013,thePewResearchCenterposedthefollowingquestionabout“drones”(inthiscaseROWS)toparticipantsintwentycountries:

1. DoyouapproveordisapproveoftheUnitedStatesconductingmissilestrikesfrompilotlessaircraftcalleddronestotargetextremistsincountriessuchasPakistan,YemenandSomalia?

Thisquestionishighlyapplication-specificandtechnology-neutral,andwecanpredictthatitalsoinducesmoralemotionalresponses,andanthropomorphicframingeffects,inparticipantsbecauseofthespecificmentionoftheUSasattacker,andofextremistsastargets.Inthisquestion,thespecificuseofdronesincarryingouttheobjectivecouldquicklyfadeintothebackground.WesuspectitrevealslessaboutpublicopiniononROWS,andmuchmoreaboutpeople’ssympathiestowardsUSforeignpolicyand“extremistsincountriessuchasPakistan,YemenandSomalia”.HuffingtonPost–YouGovPollsTwoseparateHuffingtonPost/YouGovOmnibussurveyswereadministeredinearly2013inordertogaugepublicopinionontheuseof“drones”(inthiscaseROWS).Thequestionsreadasfollows(Swanson2013):YouGovOmnibusPoll,Jan10-112013:

1. DoyouapproveordisapproveoftheObamaadministrationusingdronestokillhigh-levelterrorismsuspectsoverseas?

2. DoyouapproveordisapproveoftheObamaadministrationusingdronestokillhigh-levelterrorismsuspectsoverseas,evenifthosesuspectsareAmericancitizens?

YouGovOmnibusPoll,Feb6-72013:

1. DoyouapproveordisapproveoftheObamaadministrationusingdronestokillhigh-levelterrorismsuspectsoverseas?

2. DoyouapproveordisapproveoftheObamaadministrationusingdronestokillhigh-levelterrorismsuspectsoverseas,evenifthosesuspectsareAmericancitizens?

3. DoyouapproveordisapproveoftheObamaadministrationusingdronestokillhigh-levelterrorismsuspectsoverseas,evenifinnocentciviliansmayalsobekilled?

4. WhichcomesclosesttoyouropinionaboutwhoshouldbepermittedtoorderdronestrikesagainstsuspectedterroristswhoareUScitizens?

5. WhodoyouthinkshouldberesponsibleforsettingtherulesforwhentheCIAorU.S.militarymayusedronestrikesonAmericancitizenssuspectedofbeingterrorists?

Bothofthesepollsuseapplication-specific,technology-neutral,questionswithlanguagethatlikelyinducesmoralemotionalresponsesinparticipants.WesuspectthesesurveyscontributelittletothedebateonROWSv.AWS(seetheaboveanalysesontheWashingtonPost,PublicMindandPewResearchCentersurveysformoredetail).CharliCarpenter

13

AspartofadifferentYouGovOmnibussurvey,CharliCarpenter(2013,2014)posedaseriesofquestionstotrytogaugedifferencesinopinionbetweenROWSandAWS.CarpenterwasexplicitlyinterestedinusingherstudytohelpsatisfytherequirementssetoutintheMartensClause.Sheposedthefollowingtwoquestions:

1. Dronesareremotelypilotedbyahumancontroller,butseveralnationsaredevelopingroboticweaponsthatcanindependentlymaketargetingandfiringdecisionswithoutahumanintheloop.Howdoyoufeelaboutthetrendtowardusingcompletelyautonomous[roboticweapons/lethalrobots]inwar?

2. Somenon-profitcitizengroupshavelaunched[acampaigntobantheuseoffullyautonomousweapons/theCampaigntoStopKillerRobots],whichseeksaglobaltreatyrequiringhumaninvolvementinalldecisionstotakehumanlife.Towhatextentwouldyousupportoropposesuchacampaign?

Carpenterrandomlyvariedthelanguageinbothquestions,tocreatetwotreatmentgroups.Onegroupwaspresentedwithaversionofthequestionscontainingthe“roboticweapons/fullyautonomousweapons”wordingpair,whiletheothergrouprespondedtoaversionofthequestionscontainingthe“lethalrobots/killerrobots”wordingpair.Thesevariationsweremeanttotestifparticipantshadadifferentresponsebasedonthenatureoftherobotdescriptionsbeingused.Thisstudydesignchoiceisusefulinitsabilitytoexposebiasesthatmightbeintroducedduetoanthropomorphicframing(castingtherobotas“killer”),andmoralemotionalpriming(amoralaversionto“killing”).However,thedifferenceinwordinghadnoeffectontheresults.Weinterpretthisfindingasanindicationthattherewerelikelynosignificantdifferencesinmoralemotionalresponsesbetweentreatmentgroups.Perhapsthisfindingisnottoosurprising,sincetheword“killer”isburiedinthelargerphrase“TheCampaigntoStopKillerRobots,”andtheword“lethal”canbeusedinmorallyneutralcontexts(e.g.“thelethaldoseofsubstancexis…”).Question(1)isapplication-neutraland,assuch,itcontainsapreambletoexplainthetechnologytoparticipants.Asdiscussedaboveinrelationto(A),questionsthatareapplication-neutralcannotbeansweredadequatelyunlesstheparticipanthasanunderstandingofthetechnologyinquestion,hencetheneedforapreamble,albeitashortoneinthiscase.Asidefromthevariationsinwordingbetweentreatmentgroups,therearenoobviousmoralpsychologicalbiasingworriesraisedbythewordingof(1)or(2).WeconsiderthissurveywelldesignedandusefulinthecontextofthedebateoverROWSandAWS,atleastinsofarasitavoidsapplication-specific,technology-specificlanguage,andalsoavoidsanyobviousmoralpsychologicalbiasingtraps.MichaelCHorowitzIntwoseparatebutrelatedpublicopinionsurveysconductedin2015,Horowitz(2016a,b)claimstohaveimprovedonCarpenter’sstudy,whichhecriticizesforaskingquestionsabout

14

AWS“inavacuum”.Hisargumentis,first,thataskingquestionsaboutROWSorAWSdirectly“makesithardtodistinguishsupportoroppositiontoweaponsingeneralfromautonomousweaponsinparticular”(Horowitz2016a,4).Furthermore,askingquestionsaboutAWSwithoutreferenceto“particularscenariosandcontextsforusageand/ordevelopment”,hesays,mightinduceTerminatorBias,sinceparticipantsmightimagine“TheTerminator,TheMatrix,orotherportrayalsinthemedia”whenresponding(Horowitz2016a,4).Questionsinavacuum,heconcludes,aregoodforestablishingabaseline,butyouneedtointroduceparticularusescenarios(applications)inordertoovercometheseweaknesses.InhisstudiesheaskedabaselinequestionsimilartoCarpenter’s:

1. BaselineCondition:Dronesareremotelypilotedbyahumancontroller,butautonomousweaponsystemsareroboticsystemsthat,onceactivated,canindependentlymaketargetingandfiringdecisionswithoutahumanintheloop.WouldyouapproveordisapproveoftheUnitedStatesdevelopingautonomousweaponsystems?

Question(1)isaslightvariationonCarpenter’squestion(seeabove),subjecttoasimilaranalysis.However,heimmediatelyfollowswithseveralquestions,eachoneintroducingparticularscenarios.Forexample,heposesthefollowingadditionalquestion,whichhecallsthe“Protect+MoreEffectiveCondition”(Horowitz2016a):

2. Protect+MoreEffectiveCondition:Dronesareremotelypilotedbyahumancontroller,butautonomousweaponsystemsareroboticsystemsthat,onceactivated,canindependentlymaketargetingandfiringdecisionswithoutahumanintheloop.WouldyouapproveordisapproveoftheUnitedStatesdevelopingautonomousweaponsystemsif:

a. AutonomousweaponsystemswouldbeusedtoprotectUSmilitarypersonnelonnavalshipsandmilitarybasesfromattacksbyforeignmilitariesandmilitantgroups

b. AutonomousweaponsystemswouldbemoreeffectivethanotheroptionsatprotectingUSmilitarypersonnelonnavalshipsandmilitarybasesfromattackbyforeignmilitariesandmilitantgroups

Question(2)(a)isanapplication-specific,technologyneutralquestion,subjecttothesamecriticismspreviouslymentioned.Italsocaststhetechnologyas“protector”,makingitsusceptibletoanthropomorphicframingeffects,andsubjectingittocriticismsalreadymentioned.Question(2)(b)hasthesamefeaturesas(2)(a),withtheadditionofatechnology-specificfeature:theAWSbeingmoreeffectivethanallotherweaponoptions.Carpenter(2016)criticizesHorowitzfortheadditionofthisconditionclaimingitbiasesthestudyinfavorofkillerrobots,sincealloftheconditionsalignwithprosofAWS.Horowitz’(2016b)firstexperimentofthetwoincludesthefollowingconditions:theonestatedin(2)(b);an“Attack+MoreEffectiveCondition”;an“Attack+NotMoreEffectiveCondition”;anda“Protect+NotMoreEffectiveCondition”.AsCarpenter(2016)argues,an“objective‘contextual’”studywouldincludemany

15

moreconditions,includingscenariosthatprimeparticipantstothinkoftheconsofthetechnologythathavebeenwidelydiscussedintheliteratureonbanningAWS.ThoughamorebalancedcontextualstudymighthavetheeffectsCarpenterenvisions,wefeelthatcontextualstudiesaregenerallyproblematicfromthestartsincetheyaresusceptibletotoomanybiasingeffects,suchasmoralpsychologicalbiasingeffects.Thereisanimportantpointworthmentioninghere.ThoughweagreewithCarpenter’sassessmentofthebiasinHorowitz’choiceofconditions,wedonotthinkallofthefeaturesofhisstudyconditionsareasproblematicasCarpenterclaims.Onetechnology-specificcondition—AWSbeingmoreeffectivethanallothermilitaryoptions—escapessomecriticism.Recallthatatechnology-specificfeatureprovidesinformationthatcanfocustheparticipant’sattentiononaspectsofatechnologynotsharedbyothertechnologies.Statingthatatechnologyismoreeffectivethanallothermilitaryoptionsisafeaturethatdistinguishesitfromthoseothertechnologies,inawaythatbeing“nomore”,or“lesseffectivethanallothermilitaryoptions”donot.Furthermore,statingthatatechnologyiseither“nomore”or“lesseffectivethanallothermilitaryoptions”couldhelptofocusparticipantsonafeaturethatismorallyrelevantinthecontextoftheMartensClause:thekindoffeaturethathasconsequentialistrelevance.However,moreresearchisnecessarytounderstandifthe“nomore”or“lesseffective”conditionswouldhavethiseffect.OneimportantdifferencebetweenCarpenter’sandHorowitz’studiesisthegeneralresearchmethodsapplied:Carpenter’s(2013,2014)studyincludedaqualitativecomponentinwhichsheaskedparticipantswhytheylikedordislikedtheideaofautonomousweapons,whileHorowitz’wasstrictlyquantitative.ThisisasignificantdifferencethatCarpentermentionsinhercritiqueofHorowitz’work.Aqualitativeapproachusingopen-endedquestionsallowsparticipantstodefinethescenariosandcontextsthatarerelevanttothemoraldiscussion(Carpenter2016).Assuch,Carpenterarguesthataqualitativestudydesignprovidesamoredirectroutetothepublicconscience,whichistheheartoftheMartensClause.Moon,Danielson,andVanderLoosIncontrasttotheabovementionedpollandsurveyquestionsthataredesignedtocollectquantitativemetricsonthepublicperceptionofROWSandAWS(withthenotedexceptionofCarpenter2013,2014),therehasbeenanefforttogetahybridofquantitativeandqualitativeresponsesusingasimple,yetuniquesurveyplatform.Moon,Danielson,andVanderLoos(Moonetal.,2012)conductedastudywheretheparticipantswereprovidedwithapreambleaboutaparticulartechnologyinquestion,followedbyaquestiontheycouldanswerwitha“Yes”,“No”,or“Neutral”responseaswellasaqualitative,text-basedreasonsfortheirresponse.Theparticipants—insteadofcontributingreasonsfortheirresponse—alsohadtheoptiontovoteforanexistingresponsethatotherparticipantshavecontributedtothediscussion.Inaseriesofninequestionspertainingtovariousroboethicsissues,thefirstandsecondquestionsconcernedROWSandAWStechnologiesrespectively.

16

ThepreamblegivenforbothROWSandAWSquestionswerefromthesamereport(Linetal.2008).FortheROWSquestion,thepreambleandthequestionread:

“ThePredatorisaremotecontrolledaerialrobot.“Theserobotshavedualapplications:theycanbeusedforreconnaissancewithoutendangeringhumanpilots,andtheycancarrymissilesandotherweapons.…PredatorsareusedextensivelyinAfghanistan.TheycannavigateautonomouslytowardtargetsspecifiedbyGPScoordinates,butaremoteoperatorlocatedinNevada(orinGermany)makesthefinaldecisiontoreleasethemissiles.ShouldremotecontrolledPredatorsbearmedwithlethalweaponsincombat?”

Subsequently,theparticipantsweregivenaquotationfromareportonAWS:

“AproposedmodificationofthePredatorremotecontrolledaircraftwouldallowittoselecttargetsautonomously,withouttheinterventionofhumanoperators.“Thetechnologyto(responsibly)createfullyautonomousrobotsisnearbutnotquiteinhand.”ShouldfullyautonomouslethallyarmedPredatorsbedeveloped?”

Whiletheresultsofthisstudyissubjecttoorderbias(thequestionorderwasnotrandomized),itisalsosubjecttobiasinthelanguageusedinthepreamble—inparticular,thementionof“withoutendangeringhumanpilots”anditscontrastwiththedescriptionofthetechnologybeingusedinAfghanistan.TheintroductiontoAWS,ontheotherhand,remainsneutralinitswording.Giventhequestion’srelianceonthepreambleofthepreviousquestion,itislikelythattheparticipantsassumedthesameapplicationcontextinrespondingtotheAWSquestion.Keepingthebiasingeffectthatpotentiallyappliestothisstudy,though,themappingofquantitativeresponseswithqualitativereasonsgivenbytheparticipantsprovidedaricherunderstandingofthemoralpsychologyatworkwhenparticipantswerethinkingthroughthetwodifferenttechnologies.Forexample,theauthorsfoundthattheparticipantsweregivingmuchmorerationalandconsequentialistreasonsforsupportingorrejectingROWS.Incontrast,thesameparticipantswhowereinsupportofROWSforfunctionalorpracticalrationales(e.g.,effectivenessofthetechnology,considerationforphysicalsafetyofsoldiers)shiftedtostronglynormativereasoningwhenrejectingAWS:thathumansshouldalwaysmakelife/deathdecisions.Suchashiftinthenatureofqualitativeresponses—thoughanalyzequalitativesurveysaddscostoverquantitativestudies—canbepowerfulinrevealingnuanced,yetsignificantdifferencesnatureofweaponstechnologiescanhaveonparticipants’moralpsychology.Theparticularuniquesurveyplatformitselfalsohaslimitations,thoughadiscussionofthatisbeyondthescopeofthispaper.Nonetheless,thequalitativesetofreasonscollectedfromthisstudyhelpedinformthedesignofalargerstudyconductedbytheOpenRoboethicsinitiative.OpenRoboethicsinitiative(ORi)

17

ORi(2015)conductedaninternationalonlinepublicopinionpollin2015togaugepublicopinionontheuseanddevelopmentofROWSandAWS.ThesurveycontainedsixquestionsaboutROWSandAWS,andeachquestionwasaccompaniedbyabriefpreambleremindingparticipantsoftheterminology.Thepreambleread(ORi2015):

Terminology:ROWS(RemotelyOperatedWeaponsystems):weaponizedsystemsinwhichapersoninaremotelocationmakesthedecisiontouselethalforce.LAWS(LethalAutonomousWeaponsystems):weaponizedsystemsinwhichthesystem,withoutrequiringhumanintervention,makesthedecisiontouselethalforce.

Thequestionsreadasfollows:

1. ConsiderLethalAutonomousWeaponsystems(LAWS)technologybecomingavailabletoyourcountryinthenearfuture.Ifyourcountrygoestowaragainstanothercountry,wouldyousupporttheuseofLAWSoverRemotelyOperatedWeaponsystems(ROWS)?WithLAWS,thesystemwillmakethedecisiontouselethalforcewithouthumaninput,whereasROWSrequiresapersontomakethedecisionforit.

2. Hypothetically,ifLethalAutonomousWeaponsystems(LAWS)technologybecomesavailabletoacountrythatisattackingyourcountry,wouldyouratherbeunderattackbyLAWSthanRemotelyOperatedWeaponsystems(ROWS)?WithLAWS,thesystemwillmakethedecisiontouselethalforcewithouthumaninput,whereasROWSrequiresapersontomakethedecisionforit.

3. IfcertaintypesofLAWSaretobeinternationallybannedfromdevelopmentanduse,whattypesofLAWSdoyoufeelmoststronglyshouldbebanned?(Chooseone)

a. air(e.g.,drones,fighterpilots)b. sea(e.g.,ships,submarines)c. land(e.g.,infantry)d. alloftheaboveshouldbebannede. noneoftheaboveshouldbebanned

4. WhatdoyouthinkisthemainreasonforsupportingthedevelopmentanduseofLAWSinbattlefields?(Chooseone)

a. CostofwarwillbecheapertouseLAWSthanROWSb. Autonomousmachineswillmakemoreethicallife/deathdecisionsthanhumansc. LAWSwillsavehumanmilitarypersonnelfrompsychologicalharmofwar,suchaspost-

traumaticstressdisorder(PTSD)d. LAWSwillsavehumanmilitarypersonnelfromphysicalharmofware. DevelopmentofLAWSwillleadtothedevelopmentofuseful,non-militarytechnologiesf. TherearenovalidreasonsfordevelopingandusingLAWSoverROWSg. Other__________________

5. WhatdoyouthinkisthemainreasonforrejectingthedevelopmentanduseofLAWSinbattlefields?(Chooseone)

a. Theriskofthetechnologyfallingintothewronghandsistoobigb. Humansshouldalwaysbetheonetomakelife/deathdecisions(i.e.,itiswrongfor

machinestomakethedecision)

18

c. Itisuncertainwhowillberesponsiblewhenthingsgowrongd. LAWSwillkillmorelivesthanitwillsavee. ItisdoubtfulthatLAWStechnologyofthenearfuturewillbetechnicallyrobustand

reliableenoughtobetrustedf. TherearenovalidreasonsforrejectingthedevelopmentanduseofLAWSoverROWSg. Other_________________

6. Ingeneral,shouldLAWSthatmakelife/deathdecisionswithouthumanoperatorbedevelopedorused?

a. Yes,LAWSshouldbedevelopedandusedforbothdefenseandoffensepurposesb. Yes,butLAWSshouldbedevelopedandusedfordefensepurposesonlyc. LAWSshouldbedeveloped,butneverusedd. No,LAWSshouldnotbedevelopedandusede. Other:____________

Questions(1)and(2)areapplication-neutralquestions,containingnoobviousmoralpsychologicalbiasingtraps.Thosequestionswereaskedpriortotheothersspecificallytoavoidanybiasingeffectsthatcouldspilloverfromthecontextualoptionscontainedin(4)-(5).Questions(4)-(5)containspecificmentionofvalue-ladenaspectsassociatedwiththedevelopmentanduseofAWS,whichcouldresultinmoralpsychologicalbiasingeffects.Thesequestionswerealsoalwaysaskedpriorto(6),whichcouldresultinspillovereffectsfromanybiasintroducedin(4)-(5).However,thevalue-ladenaspectsmentionedin(4)-(5)areintendedtoelicitrationaljustificationsforusing/banningthetechnology.Thoughaqualitativedesignapproachwouldhavebeenmoreeffective,asCarpenter(2016)pointsout,thispollwasintendedforaninternationalaudienceand,assuch,practicaltranslationlimitationseliminatedthepossibilityofopen-endedquestions.Aquantitativeapproachrequiressomementionofthepossibleanswers,andthesechoicesrepresentargumentsfeaturedintheAWSliterature(e.g.PAX2014).Ideally,thesequestionsshouldbeaskedlasttoavoidbiasingeffects.MoshkinaandArkinAsadescriptive-explanatorystudy,thescopeofMoshkinaandArkin’s(2007)surveywasextensive,andwidestacrossthesimilarsurveysonthetopicofAWS.Forexample,MoshkinaandArkinexaminedthespecificrolesweaponizedroboticsystemscouldtakeon,suchascrowd/mobcontrol,sentry,prisonguard,hostagerescue,reconnaissance,directcombat.Demographicgroupsalsoincludedroboticsresearchers,policymakers,themilitary,andthegeneralpublic.Theyalsoexploredthestakeholderperceptionacrossdifferentsituations:openwarfarewithwaronforeignterritory,waronhometerritory,covertoperationsonforeignterritory,andcovertoperationsonhometerritory.However,oneaspectoftheirstudydeservesmentionbecauseitisuniquetotheirstudyandappliesgenerallytotheirpreamble,andtomanyofthequestionstheyasked.Asageneralapproach,foreachpointofinterestMoshkinaandArkin(2007)askthreeconsecutivequestions:thefirstdealswithhumansoldiersperformingtaskX,theseconddealswithhumansandrobotsasextensionsofhumansperformingtaskX,thethirddealswith

19

autonomousrobotsperformingtaskX.Hereisthepreamblefollowedbyanexampleofasetofthreesuchquestions:Preamble:

Setofthreequestions:

1. Towhatextentdoyouagreeordisagreethatitisacceptableforahumansoldiertotakeahumanlife,inamannerconsistentwiththeexistinglawsofwar,inEachofthefollowingsituations:[listofcases]

2. Towhatextentdoyouagreeordisagreethatitisacceptableforarobotactingasanextensionofahumansoldiertotakeahumanlife,inamannerconsistentwiththeexistinglawsofwar,inEachofthefollowingsituations:[listofcases]

3. Towhatextentdoyouagreeordisagreethatitisacceptableforanautonomousrobottotakeahumanlife,inamannerconsistentwiththeexistinglawsofwar,inEachofthefollowingsituations:[listofcases]

Inallsuchcaseswefeelthatthefirstquestioncouldbiasparticipantsbyframingtheissuesinthecontextofhumansoldiers,thusinducingananthropomorphicresponse.Indeed,thethreequestionsstepparticipantsthroughagradualdistancing,keepingthehumansoldierinviewuntilthethirdquestion.Bythetimeparticipantsencountertheautonomousrobottheycouldverywellbethinkingofitmorelikeahumansoldierthantheywouldotherwisehave.Ofcourse,moreworkwouldneedtobedonetotesttheseconcerns.ConcludingremarksItwilltakesometimetosettlethedebateovertheethicsandgovernanceofautonomousweaponsystems.Attemptstogaugethepublicconscience,inthespiritoftheMartensClause,willplayaroleinsettlingthatdebate.ThoughtherearenumerouspublicpollsandsurveysthathavebeenconductedtomeasurethepublicbeliefsaboutROWSandAWS,manyofthepollsaresubjecttobiaseseitherwellknowninthesurveydesignliterature(Carpenter2016),ordiscussedinthispaper,namely,application-specificandtechnology-neutralquestioning,moralemotionalpriming,andanthropomorphicframing.Itisimportanttoengagethepublic,butwemustbeawareofthepitfallshighlyapplicabletopoll/surveyquestionsonthistopic.Onlythen,

20

wewillbeabletohaveameaningfuldiscussionwiththepublicandeffectivelyinformgovernancedecisionsonAWS.ItisalsoworthnotingthatallbuttwoofthesurveysexaminedherewereconductedintheU.S.alone.9AWSareaninternationalissue.Theireffectswillspilloverborders,affectingdifferentpublicstodifferentdegrees.Itisthereforecrucialthatwelookinternationallywhenconductingpublicpollingonthis,andother,armscontrolissues.Thedataweusetoframeconclusionsaboutwhichweaponsareacceptable,andwhicharenot,oughttoberepresentativeofthosewhostandtoconfrontthoseweaponsinconflict.AWSpromisestoleaveveryfewpeopleuntouched.Therefore,thepublicconscience,asmentionedintheMartensClause,cannotbeinterpretedasapplyingonlytosinglenationstates.Researchersshould,wheneverpossible,worktodesigntheirstudieswell,andgatherdatafrombeyondtheirownborders,tohelppaintasaccurateapictureoftheglobalmorallandscapeasispossible.

9TheexceptionsarePewResearchCenter(2013),andtheOpenRoboethicsinitiative(2015).

21

References

Article36.(2015).KillingByMachine:KeyIssuesforUnderstandingMeaningfulHumanControl.Online:http://www.article36.org/autonomous-weapons/killing-by-machine-key-issues-for-understanding-meaningful-human-control/

Arkin,R.C.(2015).“WarfightingRobotsCouldReduceCivilianCasualties,soCallingforaBanNowIsPremature,”IEEESpectrum(5Aug).Online:http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/autonomous-robotic-weapons-could-reduce-civilian-casualties

Bartneck,C.,vanderHoek,M.,Mubin,O.,AlMahmud,A.(2007).“’Daisy,Daisy,GiveMeYourAnswerDo!’:SwitchingOffaRobot.”HRI’07Proceedings:217-222.

Brant,T.(2016).“U.S.MilitaryEyesAutonomousWeapons.”PCMAG.com(4Mar.).Online:http://www.pcmag.com/article2/0,2817,2500351,00.asp

Carpenter,C.(2013).“HowDoAmericansFeelAboutFullyAutonomousWeapons?”DuckofMinerva.com(19Jun.).Online:http://duckofminerva.com/2013/06/how-do-americans-feel-about-fully-autonomous-weapons.html

---.(2014).“Who’safraidofkillerrobots?(andwhy).”TheWashingtonPost(30May).Online:https://www.washingtonpost.com/news/monkey-cage/wp/2014/05/30/whos-afraid-of-killer-robots-and-why/

---.(2016).“How(Not)toMeasurethe‘PublicConscience’.”DuckofMinerva.com(7Mar.).Online:http://duckofminerva.com/2016/03/how-not-to-measure-the-public-conscience.html

Cassese,A.(2000).“TheMartensClause:HalfaLoaforSimplyPieintheSky?”EuropeanJournalofInternationalLaw11(1):187-216.

Cillizza,C.(2013).“TheAmericanPublicLovesDrones.”Washingtonpost.com.Online:https://www.washingtonpost.com/news/the-fix/wp/2013/02/06/the-american-public-loves-drones/Darling,K.(2015).“Who’sJohnny?”

Devlin,H.(2016).“KillerRobotsandDigitalDoctors:HowCanWeProtectSocietyfromAI?”TheGuardian(16Mar.).Online:http://www.theguardian.com/technology/2016/mar/15/killer-robots-driverless-cars-alphago-and-the-social-impact-artificial-intelligence.

Duffy,B.R.(2003).“AnthropomorphismandtheSocialRobot.”RoboticsandAutonomousSystems42:177-190.

22

Evelith,R.(2014).“SoWhatExactlyIsAKillerRobot?”TheAtlantic(20Aug.).Online:http://www.theatlantic.com/technology/archive/2014/08/calling-autonomous-weapons-killer-robots-is-genius/378799/

Gibbs,S.(2015).“Musk,WozniakandHawkingUrgeBanOnWarfareAIandAutonomousWeapons.”TheGuardian(27July).Online:http://www.theguardian.com/technology/2015/jul/27/musk-wozniak-hawking-ban-ai-autonomous-weapons

Haidt,J.,(2013).TheRighteousMind:WhyGoodPeopleareDividedbyPoliticsandReligion.(VintageBooks).

Horowitz,M.,Scharre,P.(2015).MeaningfulHumanControlinWeaponSystems:APrimer.CentreforaNewAmericanSecurity.Online:http://www.cnas.org/sites/default/files/publications-pdf/Ethical_Autonomy_Working_Paper_031315.pdf

Horowitz,M.C.(2016a).“PublicOpinionandthePoliticsoftheKillerRobotsDebate.”Research&Politics.Online:http://rap.sagepub.com/content/3/1/2053168015627183

---.(2016b).“OnlineAppendixforPublicOpinionandthePoliticsoftheKillerRobotsDebate.”Research&Politics.Online:http://rap.sagepub.com/content/sprap/suppl/2016/02/16/3.1.2053168015627183.DC1/Supplementary_material.pdf

HumanRightsWatch(2012)Losinghumanity:Thecaseagainstkillerrobots.Online:http://www.hrw.org/sites/default/files/reports/arms1112_ForUpload.pdf

---.(2015).MindtheGap:TheLackofAccountabilityforKillerRobots.Online:https://www.hrw.org/report/2015/04/09/mind-gap/lack-accountability-killer-robots.

InternationalCommitteeoftheRedCross(ICRC).(2009).“TheGenevaConventionsof1949:OriginsandCurrnetSignificance.”ICRC.Online:https://www.icrc.org/eng/resources/documents/statement/geneva-conventions-statement-120809.htm

---.(2015).AutonomousWeaponSystems:IsItMorallyAcceptableforaMachinetoMakeLifeandDeathDecisions?(13Apr.).Online:https://www.icrc.org/en/document/lethal-autonomous-weapons-systems-LAWS

InternationalJusticeResourceCentre.(2016).“InternationalHumanitarianLaw.”Online:http://www.ijrcenter.org/international-humanitarian-law/

Johansson,G.(1973).“Visualperceptionofbiologicalmotionandamodelforitsanalysis.”Perception&Psychophysics14(2):201–211.

23

Kahn,P.H.etal.(2011).“TheNewOntologicalCategoryHypothesisinHuman-RobotInteraction.”Proceedingsofthe6thInternationalConferenceonHuman-RobotInteraction:159-160.

---.(2012).“DoPeopleHoldaHumanoidRobotMorallyAccountablefortheHarmItCauses?”HRI’12Proceedings:33-40.

Kreps,S.(2014).“FlyingUndertheRadar:AStudyofPublicAttitudesTowardsUnmannedAerialVehicles.”ResearchandPolitics(April-June):1-7.

Lerner,J.S.,Goldberg,J.H.,Tetlock,P.E.(1998).“SoberSecondThought:TheEffectsofAccountability,Anger,andauthoritarianismonattributionsofresponsibility.”PersonalityandSocialPsychologyBulletin24:563-574.

Lin,P.,Bekey,G.,Abney,K.(2008).AutonomousMilitaryRobots:Risk,Ethics,andDesign.AreportcommissionedbyU.S.DepartmentofNavy/OfficeofNavalResearch.Online:ethics.calpoly.edu/ONR_report.pdf.

McDonald,G.(2016).“ArmedRobotsCouldDecidetoPullTrigger.”DiscoveryNews(8Mar.).Online:http://news.discovery.com/tech/robotics/armed-robots-could-decide-to-pull-trigger-160307.htm

Mikula,G.(1986).“TheExperienceofInjustice:TowardaBetterUnderstandingofItsPhenomenology.”InH.W.Bierhoff,R.L.Cohen&J.Greenberg(Eds.)JusticeandSocialRelations.(NewYork:Plenum):103-124.

Millar,J.(2015).“MeaningfulHumanControlandDual-UseTechnologies.”UNInformalMeetingofExpertsonLethalAutonomousWeapons(LAWs)undertheframeworkoftheConventiononCertainConventionalWeapons(CCW),April13-17,2015,Geneva,Switzerland.Online:http://www.unog.ch/80256EDD006B8954/(httpAssets)/F483D421E67D230FC1257E2F0033E690/$file/Jason+Millar+-+Meaningful+Human+Control+and+Dual-Use+Technology.pdf

Moon,A.,Danielson,P.,VanderLoos,H.F.M.(2012).“Survey-BasedDiscussionsonMorallyContentiousApplicationsofInteractiveRobotics.”InternationalJournalofSocialRobotics4(1):77-96.

Moshkina,L.,Arkin,R.C.(2007).LethalityandAutonomousSystems:SurveyDesignandResults.MobileRobotLaboratory,GeorgiaInstituteofTechnology.Online:https://smartech.gatech.edu/handle/1853/20068

OpenRoboethicsinitiative(ORi).(2015).“TheEthicsandGovernanceofLethalAutonomousWeaponsystems:AnInternationalPoll.”OpenRoboethicsinitiative(9Nov.).Online:http://www.openroboethics.org/wp-content/uploads/2015/11/ORi_LAWS2015.pdf

24

Paulhus,D.L.(1991).“Measurementandcontrolofresponsebiases.”InJ.P.Robinsonetal.(Eds.)Measuresofpersonalityandsocialpsychologicalattitudes.(SanDiego:AcademicPress).

Parkon,S.(2015).“KillerRobots:TheSoldiersthatNeverSleep.”BBC(16Jul.).Online:http://www.bbc.com/future/story/20150715-killer-robots-the-soldiers-that-never-sleep.

PAX.(2014).DeadlyDecisions:8ObjectionstoKillerRobots.Online:http://www.paxforpeace.nl/stay-informed/news/stop-killer-robots-while-we-still-can

PewResearchCenter.(2013).“U.S.UseofDrones,UnderNewScrutiny,HasBeenWidelyOpposedAbroad.”Pewresearch.org.Online:http://www.pewresearch.org/2013/02/06/u-s-use-of-drones-under-new-scrutiny-has-been-widely-opposed-abroad/

---.(2016).U.S.SurveyResearch.Online:http://www.pewresearch.org/methodology/u-s-survey-research/.

Prinz,J.J.,Nichols,S.(2010).“MoralEmotions.”InJ.M.Doris(Ed.)TheMoralPsychologyHandbook.(Oxford:OxfordUniversityPress):111-146.

Schnall,S.,Haidt,J.,Clore,G.,&Jordan,A.(2008).“DisgustasEmbodiedMoralJudgment.”PersonalityandSocialPsychologyBulletin34:1096-1109.

Swanson,E.(2013).“DronePollFindsSupportForStrikes,WithLimits.”HuffingtonPost.com(8Feb.).Online:http://www.huffingtonpost.com/2013/02/08/drone-support-poll_n_2647051.html

Tucker,P.(2014).“EveryCountryWillHaveArmedDronesWithin10Years.”DefenseOne(6May),Online:http://www.defenseone.com/technology/2014/05/every-country-will-have-armed-drones-within-ten-years/83878/

UNIDIR.(2014).FramingDiscussionsontheWeaponizationofIncreasinglyAutonomousTechnologies.Online:http://www.unidir.org/en/publications/framing-discussions-on-the-weaponization-of-increasingly-autonomous-technologies

UniversityofMassachusettsAmherst.(2013).“NewSurveyShowsWidespreadOppositionto‘KillerRobots,’SupportforNewBanCampaign.”Newswise(19Jun.).Online:http://www.newswise.com/articles/new-survey-shows-widespread-opposition-to-killer-robots-support-for-new-ban-campaign

WashingtonPosta.(2012).http://www.washingtonpost.com/wp-srv/politics/polls/postabcpoll_020412.html(rawdata)

Weizenbaum,J.(1976).ComputerPowerandHumanReason:FromJudgmenttoCalculation.(SanFrancisco:W.H.Freeman&Company).

25

Woolley,P.,Jenkins,K.(2013).“PublicSaysIt’sIllegaltoTargetAmericansAbroadAsSomeQuestionCIADroneStrikes.”PublicMind.Online:http://www.publicmind.fdu.edu/2013/drone/