Attack under Disguise: An Intelligent Data Poisoning ...mh6ck/papers/ · Amazon Mechanical Turk (AMT)1 and CrowdFlower2, crowdsourc-ing has emerged as a popular, fast and cheap problem-solving

Attack under Disguise: An Intelligent Data Poisoning AttackMechanism in Crowdsourcing

Chenglin MiaoState University of New York

at Buffalo, NY, [email protected]

Qi LiUniversity of Illinois at

Urbana-Champaign, IL, [email protected]

Lu Su∗State University of New York


Mengdi HuaiState University of New York


Wenjun JiangState University of New York


Jing GaoState University of New York


ABSTRACTAs an effective way to solicit useful information from the crowd,crowdsourcing has emerged as a popular paradigm to solve chal-lenging tasks. However, the data provided by the participatingworkers are not always trustworthy. In real world, there may ex-ist malicious workers in crowdsourcing systems who conduct thedata poisoning attacks for the purpose of sabotage or financial re-wards. Although data aggregation methods such as majority votingare conducted on workers’ labels in order to improve data quality,they are vulnerable to such attacks as they treat all the workersequally. In order to capture the variety in the reliability of workers,the Dawid-Skene model, a sophisticated data aggregation method,has been widely adopted in practice. By conducting maximumlikelihood estimation (MLE) using the expectation maximization(EM) algorithm, the Dawid-Skene model can jointly estimate eachworker’s reliability and conduct weighted aggregation, and thus cantolerate the data poisoning attacks to some degree. However, theDawid-Skene model still has weakness. In this paper, we study thedata poisoning attacks against such crowdsourcing systems withthe Dawid-Skene model empowered. We design an intelligent at-tack mechanism, based on which the attacker can not only achievemaximum attack utility but also disguise the attacking behaviors.Extensive experiments based on real-world crowdsourcing datasetsare conducted to verify the desirable properties of the proposedmechanism.

CCS CONCEPTS• Security and privacy → Systems security; • Human-centered computing→ Collaborative and social computing;

KEYWORDSCrowdsourcing, data poisoning, expectation maximization

∗L. Su is the corresponding author.

This paper is published under the Creative Commons Attribution 4.0 International(CC BY 4.0) license. Authors reserve their rights to disseminate the work on theirpersonal and corporate Web sites with the appropriate attribution.WWW 2018, April 23–27, 2018, Lyon, France© 2018 IW3C2 (International World Wide Web Conference Committee), publishedunder Creative Commons CC BY 4.0 License.ACM ISBN 978-1-4503-5639-8/18/04.https://doi.org/10.1145/3178876.3186032

ACM Reference Format:Chenglin Miao, Qi Li, Lu Su, Mengdi Huai, Wenjun Jiang, and Jing Gao. 2018.Attack under Disguise: An Intelligent Data Poisoning Attack Mechanism inCrowdsourcing. In The 2018 Web Conference, April 23–27, 2018, Lyon, France.ACM,NewYork, NY, USA, 10 pages. https://doi.org/10.1145/3178876.3186032

1 INTRODUCTIONWith the proliferation of online crowdsourcing services such asAmazon Mechanical Turk (AMT)1 and CrowdFlower2, crowdsourc-ing has emerged as a popular, fast and cheap problem-solving para-digm for various data analysis tasks, such as image annotation [52],entity resolution [47] and sentiment analysis [33]. Through thepower of the crowd, the data requesters can obtain large amountsof data at extremely low cost. In a typical crowdsourcing system,the data requester posts the tasks that require human intelligenceonto a crowdsourcing service platform, and then a large crowdof people (usually referred to as workers) participate in the tasksthat they are interested in. In crowdsourcing systems, to reduce theerrors made by individual workers, a common practice is to queryan item (e.g., a picture or a question) to multiple workers and thenaggregate their labels on the items.

Although crowdsourcing brings substantial advantages, theopenness of the crowdsourcing systems and the potential valueof the collected data offer both opportunities and incentives formalicious parties to launch attacks. In this paper, we investigatecrowdsourcing in adversarial environments and study an importantattack form, called data poisoning. In this attack, the attacker aimsto maximize the error of the final results and render the crowd-sourcing results useless through creating or recruiting a group ofmalicious workers and letting them provide manipulated data. Thisattack goal can be easily achieved if the attacker has the capabilityof creating or recruiting an overwhelming number of maliciousworkers. However, in practice, the attacker usually has limited re-sources and he can only control a few malicious workers. In suchcases, the attack strategy plays an important role.

A naive attack strategy is to let the malicious workers alwaysdisagree with the normal workers. If some straightforward aggre-gation methods, such as majority voting, are used to aggregate thedata, this naive attack model may be the optimal choice, since ev-ery malicious worker exerts the most influence in the aggregation.

1https://www.mturk.com2https://www.crowdflower.com

Track: Crowdsourcing and Human Computation for the Web WWW 2018, April 23-27, 2018, Lyon, France

13

https://doi.org/10.1145/3178876.3186032

https://doi.org/10.1145/3178876.3186032

However, the story would become much more complicated whensome more sophisticated aggregation methods that can capture thereliability (i.e., data quality) of each worker are employed. A rep-resentative method in this category is the Dawid-Skene model [9],which has been widely adopted in crowdsourcing to aggregatethe conflicting data from different workers. In the Dawid-Skenemodel, each worker is associated with an underlying confusionmatrix, which can reflect the reliability degree of this worker. Afterthe labels are collected from the workers, the final results and theworkers’ confusion matrices are jointly estimated based on the max-imum likelihood principle. As a result, workers with low reliabilitydegrees will have low impact in the aggregation. In this case, ifan attacker adopts the aforementioned naive attack, in which themalicious workers always disagree with the normal workers, themalicious workers are very likely to be assigned a significantly lowreliability degree by the Dawid-Skene model, and thus will not beable to make any difference in the final aggregated results.

To attack a crowdsourcing system with the Dawid-Skene modelempowered, we propose an intelligent data poisoning attack mech-anism that takes into account the malicious workers’ reliabilitydegrees. In this mechanism, the malicious workers behave more “in-telligently”, i.e., try to improve their reliability degrees by agreeingwith the normal workers on some items whose values are unlikelyto be overturned. Compared with the aforementioned naive strat-egy, the proposed intelligent attack model can not only disguise themalicious workers, but also enable them to launch more effectiveattacks on the items that are more vulnerable to attack.

Towards this end, we formulate a bi-level optimization prob-lem. The objective in the optimization problem is to maximize theattacker’s utility, which is the combination of the number of thesuccessfully attacked items and the malicious workers’ reliabilitydegrees. Since the number of the successfully attacked items isdiscrete, it is hard to directly solve the optimization problem. Toaddress this challenge, a continuous and differentiable sigmoidfunction is adopted to approximate the discrete component in theobjective function. We solve the bi-level optimization problem byiteratively solving the upper-level and lower-level subproblems,which are solved by the projected gradient ascent and expectation-maximization (EM) methods, respectively.

In summary, the main contributions of this paper are:• We identify the pitfalls and challenges in attacking a crowdsourc-ing system empowered with the Dawid-Skene model, which isable to incorporate the workers’ reliability degrees into the ag-gregation procedure and thus can tolerate the naive maliciousattacks.• We design an intelligent data poisoning attack mechanism, basedon which the attacker can achieve the optimal attack goal byintelligently disguising the malicious workers’ behaviors.• Extensive experiments based on real-world crowdsourcingdatasets are conducted to verify the advantages of the proposedmechanism.

2 PROBLEM SETTINGIn this paper, we consider a crowdsourcing scenario in which acloud server and some participating workers are involved. Thecloud server is a platform which can outsource the crowdsourcingtasks to the participating workers. The crowdsourcing task is to

Figure 1: Crowdsourcing framework with malicious work-ers.

collect labels for a pool of items, each of which belongs to one oftwo possible categories (e.g., duchenne smile/non-duchenne smile;same/different; positive/negative; etc.). To ensure the quality ofthe final result, each item will be queried to multiple participatingworkers, who are the individuals that carry out the crowdsourcingtasks, and each worker will provide labels for a number of items.After collecting the labels from the participating workers, the cloudserver aggregates these labels to derive the true label of each item.

The security threats considered in this paper mainly come froman attacker who aims to attack the crowdsourcing system for mali-cious purposes. The goal of the attacker is to maximize the error ofthe derived true labels, and meanwhile disguise his malicious behav-iors so that the attack cannot be detected easily. We assume that theattacker can recruit or create multiple participating workers (calledmalicious workers) and arbitrarily manipulate their labels, but hecannot influence the behaviors of the normal workers who carry outthe crowdsourcing tasks without any malicious purpose. If there isno limitation on the ability of the attacker, he can achieve the attackgoal easily through creating a large number of malicious workers.However, in practice, the attacker usually has limited resourcesand can only recruit or create a few malicious workers. In suchcases, it is essential for the attacker to design a sophisticated attackstrategy (i.e., the labels provided by the malicious workers) suchthat the attack goal can be maximally achieved. In order to assessthe vulnerability of the crowdsourcing system in the worst case, wealso assume that the attacker has full knowledge of the aggregationmethod and the labels from normal workers. This assumption isreasonable as it is possible for the attacker to learn the labels ofnormal workers through eavesdropping the communications be-tween the cloud server and the normal workers. Figure 1 shows thecrowdsourcing framework with malicious workers.

Problem formulation. Suppose the cloud server releasesa crowdsourcing task which contains a set of items O =

o1,o2, ...,oM , and these items are queried to K normal workerswhich are represented as U = u1,u2, ...,uK . The labels providedby these normal workers are denoted as X = xkm

M,Km,k=1, in which

xkm is the label provided by worker uk for item om . For each itemom , there is a true label x∗m which is unknown a priori and needs tobe estimated by the cloud server based on the labels collected fromall the workers. We use X ∗ = x∗m Mm=1 to denote the set of true la-bels for all items. Assume that the attacker can create K ′ maliciousworkers represented as U = u1, u2, ..., uK ′. The set of labels pro-vided by all the malicious workers is denoted as X = xk

′

m M,K ′m,k ′=1,


14

andxk′

mfistheflabeflprovfidedbymaflficfiousworkeruk′fforfitemom.Ourgoaflfinthfispaperfistofindanoptfimaflattackstrategy(fi.e.,an

optfimaflX)ffromtheperspectfiveofftheattackersuchthattheattackgoaflcanbemaxfimaflflyachfieved.

3 PRELIMINARYAsaflow-costprobflem-soflvfingparadfigmutfiflfizfingthewfisdomoffcrowds,crowdsourcfinghasbeenwfideflyadoptedtocoflflectflabeflsfforvarfioustasks.Thefitemsaredfistrfibutedtomufltfipflepartficfipatfingworkers,andthentheflabeflsarecoflflectedffromthemtoestfimatethetrueflabefloffeachfitemfinthetask.Duetothevarfietyfinthequaflfityoffthepartficfipatfingworkers,fitfisacommonpractficetoqueryeachfitemtoseveraflworkersandthenaggregatethefirflabeflsfinordertogetamorereflfiabfleresuflt.Astrafightfforwardaggregatfionmethodfismajorfityvotfing.However,thfismethodcannotdfistfingufishthereflfi-abfiflfitydegreesofftheworkers.Inordertotaketheworkers’quaflfityfintoaccountandobtafinmoreaccurateresuflts,theDawfid-Skenemodefl[9]hasbeenwfideflyadoptedfincrowdsourcfingsystems.IntheDawfid-Skenemodefl,eachpartficfipatfingworkerfisassocfi-

atedwfithanunknownconffusfionmatrfixwhfichreflectstheworker’sabfiflfity(orreflfiabfiflfitydegree)whencarryfingoutthecrowdsourcfingtask.Eachdfiagonafleflementfinthfismatrfixrepresentstheproba-bfiflfitythattheworkerprovfidesthetrueflabeflfforapartficuflarfitem,whfifleeachoff-dfiagonafleflementrepresentstheprobabfiflfitythatapartficuflarwrongflabeflfisprovfided.Afftertheflabeflsarecoflflectedffromaflfltheworkers,themaxfimumflfikeflfihoodestfimatfionmethodfisadoptedtojofintflyestfimateeachfitem’strueflabeflandeachworker’sconffusfionmatrfix.Inthfispaper,weconsfiderthebfinarycase,fi.e.,weassumethat

eachfitemhasonflytwopossfibfleflabefls:0and1.BasedontheDawfid-Skenemodefl,eachworkerukprovfidesflabeflfforfitemomaccordfing

toparametersαk=Pr(xkm=1|x

∗m=1)andβk=Pr(x

km=0|x

∗m=

0),whereαkandβkarethedfiagonafleflementsfinworkeruk’sconffusfionmatrfix.Theyareaflsotreatedasuk’sabfiflfityparametersorreflfiabfiflfitydegrees.Theflargerαkandβkare,thehfighertheprobabfiflfitythatworkerukprovfidesatrueflabefl.Addfitfionaflfly,thfismodeflassumesthattheprobabfiflfitythatanfitemdrawnatrandomhastrueflabefl1fisp,whfichfisusuaflflyunknownaprfiorfi.DenoteΘ=p,αk,βk

Kk=1asthesetoffaflflthemodeflparameters.The

Dawfid-SkenemodefladoptsthemaxfimumflfikeflfihoodestfimatfionmethodtoestfimateΘ.However,duetotheflatentvarfiabflesX∗areunknownaprfiorfi,fitfishardtodfirectflyconducttheestfimatfion.Toaddresstheabovechaflflenge,theDawfid-Skenemodefladopts

theEMaflgorfithm[11]whfichcontafinsanexpectatfionstep(E-step)andamaxfimfizatfionstep(M-step).IntheE-step,thefitems’trueflabeflsarederfivedbasedontheestfimatedmodeflparametersΘ,andfintheM-step,theparametersΘarecaflcuflatedbasedonthederfivedtrueflabefls.Thedetafiflsoffthetwostepsaredescrfibedasffoflflows.E-step:Inthfisstep,themodeflparametersΘarefixed.Foreach

fitemom,wecaflcuflateωm =Prx∗m =1|XbasedontheBayes

theorem:

ωm =Prx∗m =1|X;Θ=

PrX|x∗m =1·p

PrX|x∗m =1·p+PrX|x∗m =0·(1−p)

=k∈Um α

xkmk(1−αk)

1−xkm ·p

k∈Um αxkmk(1−αk)

1−xkm ·p+ k∈Um β1−xkmk

(1−βk)xkm ·(1−p)

,

(1)

whereUmrepresentsthesetoffnormaflworkerswhoprovfideflabeflsfforfitemom.Hereωmfistheposterfiorprobabfiflfitythatthetrueflabefloffthe

fitemomfis1.WfiththecaflcuflatedΩ=ωmMm=1,theexpectedvaflue

offtheflogflfikeflfihoodffunctfioncanbeexpressedas

Q(Θ)=E[flogL(Θ;X,X∗)]=E[flogM

m=1

L(Θ;Xm,x∗m)]

=M

m=1

ωmflog[k∈Um

αxkmk(1−αk)

1−xkm ·p]

+(1−ωm)flog[k∈Um

β1−xkmk

(1−βk)xkm ·(1−p)],

(2)

whereXmrepresentsthesetoffflabeflsfforfitemom.M-step:Inthfisstep,theposterfiorprobabfiflfitfiesωm

Mm=1are

fixed.ThemodeflparametersΘareestfimatedbymaxfimfizfingtheexpectedvaflueofftheflogflfikeflfihoodffunctfionQ(Θ),andtheyareupdatedasffoflflows:

p=Mm=1ωm

M, (3)

αk=m∈Ok

ωm ·xkm

m∈Okωm

, (4)

βk=m∈Ok

(1−ωm)·(1−xkm)

m∈Ok(1−ωm)

, (5)

whereOkrepresentsthesetofffitemsquerfiedtouk.Theabovetwostepsarefiteratfiveflyconducteduntfifltheconver-

gencecrfiterfionfissatfisfied.Ffinaflfly,fiffωmfisflargerthan0.5,thetrueflabefloffthefitemomfisassfignedas1,otherwfise,fitfisassfignedas0.

4 THEINTELLIGENTATTACKMECHANISMInordertoachfievetheattackgoaflasmuchaspossfibfle,fitfisessentfiaflffortheattackertofindanoptfimaflattackstrategywfiththeflfimfitedresources(fi.e.,thenumberoffcreatedorrecrufitedmaflficfiousworkersandthenumberoffquerfiedfitems).WefirstfinvestfigatetheDawfid-Skenecrowdsourcfingmodeflundertheadversarfiaflenvfironmentfinsectfion4.1,andthendfiscusshowtodesfignanoptfimaflattackstrategyffromtheperspectfiveofftheattackerfinsectfion4.2.

4.1 Dawfid-SkeneCrowdsourcfingModeflwfithMaflficfiousWorkers

Intheadversarfiaflenvfironment,themaflficfiousworkersmaybflendfintothecrowdsourcfingsystemandprovfidemanfipuflatedflabeflstothecfloudserverfinordertodfistortthefinaflaggregatedresuflts.Inthfissectfion,wedecomposethepartficfipatfingworkersfintonormaflandmaflficfiousones,andfinvestfigatethereflatfionshfipbetweenthefinaflaggregatfionresufltsandtheflabeflsprovfidedbymaflficfiouswork-ers.Pfleasenotethatthecfloudserverfinthecrowdsourcfingsystemcannotdfifferentfiatethetwotypesoffpartficfipatfingworkerswhenaggregatfingthecoflflectedflabefls.Asdescrfibedfintheprobflemsettfing,weassumethattheattacker

createsK′maflficfiousworkerstoconductthedatapofisonfingattacksagafinstthecrowdsourcfingsystem.Theattackercannotfinfluence

Track: Crowdsourcfing and Human Computatfion ffor the Web

thebehavfiorsoffthenormaflworkers,buthecanarbfitrarfiflyma-nfipuflatetheflabeflsoffmaflficfiousworkers.Wedenotetheabfiflfity

WWW 2018, Aprfifl 23-27, 2018, Lyon, France

15

parametersoffmaflficfiousworkeruk′fintheDawfid-Skenemodeflas

αk′=Pr(xk′m =1|x

∗m =1)andβk′=Pr(x

k′m =0|x

∗m =0).We

useΘ=p,αk,βkKk=1,αk′,βk′

K′

k′=1todenotethesetoffthe

modeflparametersandαk,βkKk=1aretheabfiflfityparametersoff

thenormaflworkers.SupposeXfisthesetofftheflabeflsprovfidedbyaflflthepartficfipatfingworkers,fincfludfingthenormaflandmaflficfiousones.TheE-stepandM-stepfintheDawfid-Skenemodeflaffterdatapofisonfingattackscanbedescrfibedasffoflflows:

E-step:Foreachfitemom,wecaflcuflateωm =Prx∗m =1|X

basedontheBayestheorem:

ωm =Prx∗m =1|X;Θ

=PrX|x∗m =1·p

PrX|x∗m =1·p+PrX|x∗m =0·(1−p)

=Am1

Am1+Am0,

(6)

where

Am1=k∈Um

αxkmk(1−αk)

1−xkm ·

k′∈Um

αxk′mk′(1−αk′)

1−xk′m ·p (7)

Am0=k∈Um

β1−xkmk

(1−βk)xkm ·

k′∈Um

β1−xk

′m

k′(1−βk′)

xk′m ·(1−p).

(8)

HereweuseUmtodenotethesetoffmaflficfiousworkerswhoprovfideflabeflsfforfitemom.ωmrepresentstheposterfiorprobabfiflfitythatthetrueflabeflofffitemomfis1affterthedatapofisonfingattacks.M-step:Inthfisstep, wefixthetheposterfiorproba-

bfiflfitfiesωmMm=1andupdatethe modeflparametersΘ =

p,αk,βkKk=1,αk′,βk′

K′

k′=1asffoflflows:

p=Mm=1ωm

M(9)

αk=m∈Ok

ωm ·xkm

m∈Okωm

, βk=m∈Ok

(1−ωm)·(1−xkm)

m∈Ok(1−ωm)

, (10)

αk′=m∈Ok′

ωm ·xk′m

m∈Ok′ωm

, βk′=m∈Ok′

(1−ωm)·(1−xk′m)

m∈Ok′(1−ωm)

,(11)

whereOk′representsthesetofffitemsquerfiedtouk′.Theaboveequatfionsshowthatoncetheflabeflsoffnormaflwork-

ers(fi.e.,X)aregfiven,thefinaflestfimatedtrueflabeflsoffthefitems

andtheabfiflfityparameters(fi.e.,αk,βkKk=1,αk′,βk′

K′

k′=1)offthe

partficfipatfingworkersareonflydependentonthemaflficfiouswork-ers’data.Dfifferentvafluesoffthemaflficfiousworkers’flabeflscanfleadtodfifferentestfimatedresuflts.Basedonthfisffact,theattackercanconductdatapofisonfingattacksthroughcareffuflflydesfignfingthemaflficfiousworkers’flabeflssuchthatthegoaflofftheattackercanbeoptfimaflflyachfieved.

4.2 OptfimaflAttackStrategyInthfispaper,theattackerconductsthedatapofisonfingattacksfforthepurposeoffmaxfimfizfingtheerroroffthefinaflaggregatedresuflts,andatthesametfimetrfiestodfisgufisehfisattackbehavfiorsasmuchaspossfibfle.Wecanunderstandthegoaflofftheattackerfintwoaspects.Ononehand,theattackerafimstomaxfimfizethe

devfiatfionbetweentheoutputsofftheDawfid-Skenecrowdsourcfingmodeflbefforeandaffterthedatapofisonfingattacks.Inotherwords,theattackerwantstomaxfimfizethenumberoffthesuccessffuflflyattackedfitems,wherewesayanfitemfisattackedsuccessffuflflyfifftheestfimatedtrueflabeflfischangedffromoneflabefltotheotherafftertakfingthemaflficfiousworkers’flabeflsfintoaccount.Ontheotherhand,theattackerwantstodfisgufisethemaflficfiousworkersasnormaflworkersfinthecrowdsourcfingsystemsuchthattheattackbehavfiorscannotbedetectedeasfifly.Onewaytoachfievethedfisgufisefistogethfighvafluesonthemaflficfiousworkers’abfiflfityparameters(or

reflfiabfiflfitydegrees),fi.e.,αk′K′

k′=1andβk′

K′

k′=1.Sfincetheworkers

wfithflargeabfiflfityparameterswfiflflbetreatedashfigh-quaflfityworkersfintheDawfid-Skenemodefl,thecrowdsourcfingsystemthencannotdfistfingufishthemaflficfiousworkersffromthenormaflworkers.Inthfissectfion,westandontheattacker’sposfitfionanddfiscusshowtofindanoptfimaflattackstrategysothatthegoaflofftheattackercanbeachfievedasmuchaspossfibfle.SupposetheattackerfisabfletocreateorrecrufitK′maflficfious

workers,andfforeachmaflficfiousworker,thequerfiedfitemsaregfiven.Whenconductfingthedatapofisonfingattackstobreakthecrowd-sourcfingsystem,theattackerneedstofindtheoptfimaflassfignmentsfforthemaflficfiousworkers’flabefls.Anfintufitfivestrategyfisfletthemaflficfiousworkersprovfidetheflabeflwhfichfisnotflfikeflytobetruefforeachquerfiedfitem.Thfisstrategymayworkweflflwhentheaggre-gatfionmethodfismajorfityvotfing.ButffortheDawfid-Skenemodefl,fitfisnottheoptfimaflchofice,especfiaflflywhenonflyaffewmaflficfiousworkersarecreatedorrecrufited.Duetotheffactthatmaflficfiousworkersaflwaysdfisagreewfiththemajorfity,theDawfid-Skenemodeflwfiflflassfignflowabfiflfityvafluestothesemaflficfiousworkers,andcon-sequentfly,thefirfimpactwfiflflaflsobedecreased.Insuchway,themaflficfiousworkerscanbedetectedeasfiflyandtheattackmayffafiflonaflflthefitems.Thustheabfiflfityparametersoffmaflficfiousworkers

(fi.e.,αk′K′

k′=1andβk′

K′

k′=1)shoufldbetakenfintoaccountwhen

findfingtheoptfimaflattackstrategy.Inordertoaddresstheabovechaflflenge,wefformuflatethegoafl

offtheattackerastheffoflflowfingoptfimfizatfionprobflem:

maxX

M

m=1

1(x∗am x∗bm)+λK′

k′=1

(αk′+βk′) (12)

s.t. X∗a,Θ=argmaxX∗a,Θ

flogL(Θ;X,X∗a)

xk′

mM,K′

m,k′=1∈0,1

whereX∗a=x∗amMm=1denotesthesetofftheestfimatedtrueflabefls

affterthedatapofisonfingattacksandx∗bm denotestheestfimatedtrueflabeflfforfitemombefforetheattacks(fi.e.,caflcuflatedbasedontheflabeflsoffnormaflworkers).Oncethenormaflworkers’flabeflsaregfiven,x∗bm fisaconstant.Theobjectfiveffunctfioncontafinstwocom-

ponents.Thefirstcomponent,fi.e., Mm=11(x

∗am x∗bm),where1(·)

fisthefindficatorffunctfion,representsthenumberoffthesuccessffuflfly

attackedfitems.Inthesecondcomponent, K′

k′=1(αk′+βk′)fisthe

summatfionoffthemaflficfiousworkers’abfiflfityparameters,andλfisaparameterusedtotradeoffthetwocomponents.Thesummatfionoffthetwocomponentscanaflsobetreatedastheutfiflfityofftheattacker.Thefintufitfionofftheobjectfiveffunctfionfistomaxfimfizethenumberoffthesuccessffuflflyattackedfitemsandthemaflficfiouswork-


ers’abfiflfityvafluessfimufltaneousfly,wherethefirstcomponentfisthe


16

goaflofftheattackandtheflatterensuresthatthemaflficfiousworkerscannotbedetectedeasfifly.Inthfisoptfimfizatfionprobflem,theDawfid-Skenemodeflbecomesaconstrafint.Thfisfisabfi-flevefloptfimfizatfionprobflem[2].Theoptfimfizatfionovertheflabeflsoffthemaflficfiouswork-

ers(fi.e.,X)fistheupper-fleveflprobflem,andtheoptfimfizatfionover

X∗a,Θfistheflower-fleveflprobflem.IntheDawfid-Skenemodefl,thefinaflestfimatedtrueflabeflsoffthe

fitemsaredependentontheposterfiorprobabfiflfitfiesΩ=ωmMm=1

orΩ=ωmMm=1:fiffωm(orωm)fisflargerthan0.5,x

∗bm (orx

∗am)fis

assfignedas1,otherwfise,fitfisassfignedas0.Thuswecanrefformuflateoptfimfizatfionprobflem(12)asffoflflows:

maxX

M

m=1

1

21−sgn[(ωm −0.5)·(ωm −0.5)]+λ

K′

k′=1

(αk′+βk′)

s.t. Ω,Θ=argmaxΩ,Θ

flogL(Θ;X,Ω) (13)

xk′

mM,K′

m,k′=1∈0,1,

where

sgn[(ωm −0.5)·(ωm −0.5)]=

1 fiff(ωm −0.5)·(ωm −0.5)>0

0 fiff(ωm −0.5)·(ωm −0.5)=0

−1 fiff(ωm −0.5)·(ωm −0.5)<0.

(14)

Oncetheflabeflsoffnormaflworkersaregfiven,theposterfiorprobabfifl-

fityωmfforfitemomfisaconstant.ωm,αk′andβk′aredependenton

theflabeflsoffthemaflficfiousworkers(fi.e.,theattackstrategyX)andtheycanbedfifferentwhenthemaflficfiousworkersvarythefirflabefls.

Inthfisway,ωm,αk′andβk′canbeexpressedastheffunctfionsoff

XaccordfingtoEqn.(6)andEqn.(11).Thenprobflem(13)becomes:

maxX

M

m=1

1

21−sgn[(ωm −0.5)·(

Am1

Am1+Am0−0.5)]

+λK′

k′=1

(m∈Ok′

ωm ·xk′m

m∈Ok′ωm

+m∈Ok′

(1−ωm)·(1−xk′m)

m∈Ok′(1−ωm)

)


flogL(Θ;X,Ω) (15)

xk′

mM,K′

m,k′=1∈0,1.

Sfincetheobjectfiveffunctfionfinprobflem(15)fisnotcontfinuous,fitfishardtodfirectflysoflvethfisoptfimfizatfionprobflem.Inordertoaddressthfischaflflenge,weapproxfimatetheobjectfiveffunctfionfinprobflem(15)bytheffoflflowfingone:

maxX

M

m=1

1−1

1+exp[−θ(ωm −0.5)·(Am1

Am1+Am0−0.5)]

+λK′

k′=1

(m∈Ok′

ωm ·xk′m

m∈Ok′ωm

+m∈Ok′

(1−ωm)·(1−xk′m)

m∈Ok′(1−ωm)

).

(16)

Thebasficfideabehfindtheapproxfimatfionfisthatffunctfionh1(x)=12(1−sgnx)canbeapproxfimatedbyffunctfionh2(x)=1−

11+exp(−θx)

whenx∈(−1,1).Theparameterθfinh2(x)representsthesteepnessoffthecurve.Thecurvesoffthetwoffunctfionswhenθ=100areshownfinFfigure2.Wecanseeh2(x)fisagoodapproxfimatfionoffh1(x).Addfitfionaflfly,thecontfinuouspropertyoffh2(x)aflflowsustosoflvetheoptfimfizatfionprobflembasedontheobjectfiveffunctfion(16).

(a)h1(x) (b)h2(x)wfithθ=100

Ffigure2:Curvesoffh1(x)andh2(x).

Anotherchaflflengewhensoflvfingtheaboveoptfimfizatfionprobflem

fisthateacheflementfinXhasacategorficaflvaflue(0or1).Thfisfintroducesdfificufltfieswhensoflvfingtheupper-fleveflprobflem.In

thfispaper,wereflaxthevafluesofftheeflementsfinXtotherange[0,1]suchthattheoptfimfizatfionprobflemcanbesoflvedaccordfing

tothegradfient-basedmethods.Inotherwords,wetreatxk′

mastheprobabfiflfitythatmaflficfiousworkeruk′provfidesflabefl1fforfitemom.

Ffinaflfly,thevaflueoffxk′

mwfiflflbetransfformedtocategorficafldata:fiff

theprobabfiflfityfisflargerthan0.5,xk′

m fisassfignedas1,otherwfise,fitfisassfignedas0.Thentheattackerneedstosoflvetheffoflflowfingoptfimfizatfionprobflemfinordertogettheoptfimaflattackstrategy:

maxX

ff(X)=M

m=1

1−1

1+exp[−θ(ωm −0.5)·(Am1

Am1+Am0−0.5)]

+λK′

k′=1

(m∈Ok′

ωm ·xk′m

m∈Ok′ωm

+m∈Ok′

(1−ωm)·(1−xk′m)

m∈Ok′(1−ωm)

)


flogL(Θ;X,Ω) (17)

xk′

mM,K′

m,k′=1∈[0,1].

Next,wedfiscusshowtosoflvetheaboveoptfimfizatfionprobflem.Thesoflutfionweadoptedherefisatwo-stepfiteratfiveprocedure.Step1:Inthfisstep,wefirstfixtheflabeflsoffmaflficfiousworkers,

fi.e.,X,whfichareestfimatedfintheprevfiousfiteratfion.Ifffitfisthe

firstfiteratfion,theeflementsfinXcanbefinfitfiaflfizedrandomflyorbesetassomepartficuflarvaflues.Thenwesoflvetheflower-fleveflprobflemthroughconductfingtheE-stepandM-stepdescrfibedfin

sectfion4.1togettheoptfimaflparametersΩ,Θ.Pfleasenotethat

aflfltheeflementsfinXneedtobetransfformedtothecategorficaflvaflues(fi.e.,1or0)befforesoflvfingtheflower-fleveflprobflem.

Step2:Inthfisstep,wefixtheparametersΩ,ΘcaflcuflatedfinStep1,andthenadopttheprojectedgradfientascentmethodtosoflvetheupper-fleveflprobflem.Morespecfificaflfly,finfiteratfiont,we

updatexk′

masffoflflows:

xk′(t+1)m ← Proj[0,1](x

k′(t)m +st·xk′m

ff(X)) (18)

wherestfisthestepsfizefinfiteratfiontandProj[0,1](·)fisthepro-

jectfionoperatorontotherange[0,1].Thegradfientxk′mff(X)fis

caflcuflatedasffoflflows:

xk′mff(X)=−

exp(θd1d2)

[1+exp(θd1d2)]2·θd1·

∂d2

∂xk′m

(19)

+λ(ωm

m∈Ok′ωm+

ωm −1

m∈Ok′(1−ωm)

)

whered1=ωm−0.5,d2=Am1

Am1+Am0−0.5.Throughcombfinfing

wfithEqn.(7)andEqn.(8),wecancaflcuflate ∂d2∂xk

′m


as


17

∂d2

∂xk′m

=

∂Am1∂xk

′m·Am0−

∂Am0∂xk

′m·Am1

(Am1+Am0)2(20)

where

∂Am1

∂xk′m

=pk∈Um

αxkmk(1−αk)

1−xkm

k′∈Um\k′

αxk′m

k′(1−αk′)

1−xk′m ·

[αxk′mk′(1−αk′)

1−xk′m flog(αk′)−α

xk′mk′(1−αk′)

1−xk′m flog(1−αk′)],

(21)

∂Am0

∂xk′m

=(1−p)k∈Um

β1−xkmk

(1−βk)xkm

k′∈Um\k′

β1−xk

′m

k′(1−βk′)

xk′m ·

[−β1−xk

′m

k′(1−βk′)

xk′m flog(βk′)+β

1−xk′m

k′(1−βk′)

xk′m flog(1−βk′)].

(22)

Theabovetwostepswfiflflbefiteratfiveflyconducteduntfiflthecon-vergencecrfiterfionfissatfisfied.Inthfispaper,wedefinetheconver-

gencecrfiterfionas K′

k′=1Mm=1(x

k′(t+1)m −x

k′(t)m )2<δ,whfich

representsthechangeoffXfintwoconsecutfivefiteratfionsbefingfless

thanathreshofldδ.AfftertheattackergetthefinaflX,theeflements

finXwfiflflbetransfformedto0or1andthenprovfidedtothecfloudserverastheflabeflsoffthemaflficfiousworkers.ThesubmfittedflabeflsXwfiflflbetreatedastheoptfimaflattackstrategyofftheattacker.TheoptfimfizatfionprocedurefissummarfizedasAflgorfithm1.

Aflgorfithm1:OptfimaflattackagafinsttheDawfid-Skenecrowdsourcfingmodefl

Input:Thenumberofffitems:M;thenumberoffnormaflworkers:K;

thenormaflworkers’flabefls:X;thenumberoffmaflficfiousworkers:K′;thefitemsquerfiedtothemaflficfiousworkers:

Ok′K′

k′=1

Output:Theoptfimaflattackstrategy:X

1InfitfiaflfizetheoptfimaflattackstrategyX;

2repeat

3 EstfimatetheoptfimaflparametersΩ,ΘthroughconductfingtheEMaflgorfithmdescrfibedfinsectfion4.1;

4 fforeachxk′

m ∈Xdo

5 Updatexk′

m accordfingtoEqn.(18);

6 end

7untfiflTheconvergencecrfiterfionfissatfisfied;

8TransfformtheeflementsfinXto0or1;

9returnTheoptfimaflattackstrategyX;

5 ATTACKWITHLIMITEDKNOWLEDGEInordertoassessthevuflnerabfiflfityoffthecrowdsourcfingsystemfintheworstcase,weconsfidertheffuflflknowfledgescenarfiofintheabovemechanfismandassumethattheattackerhascompfleteknowfledgeofftheflabeflsffromthenormaflworkers(fi.e.,normaflflabefls)fforaflflfitems.Inffact,theproposedmechanfismcanaflsobeempfloyedtofimpflementaneffectfiveattackevenwhentheattackeronflyhasflfimfitedknowfledgeoffthefitems’normaflflabefls.SupposetheattackeronflyknowsthenormaflflabeflsfforM′(M′<

M)fitemsrepresentedasO′=o′1,o′2,...,o

′M′.Wedenotetheset

offthenormaflflabeflsffortheM′fitemsasX′=x′kmM′,Km,k=1

,whfich

fisasubsetoffX.SfincetheattackerhasnoknowfledgeoffthefitemsexceptthosefinO′,agoodchoficefforhfimfinsuchascenarfiofistofletthemaflficfiousworkersonflyprovfidemanfipuflatedflabeflsfforthefitemsfinO′andtrytomaxfimfizetheerroroffthefinaflresufltsffortheM′fitems.Inordertoachfievethegoafl,theattackercoufldtreatX′asthesurrogatedataoffXandempfloytheaboveproposedmechanfismtoderfivetheattackstrategy.Inotherwords,theattackstrategyfinsuchascenarfiocanbederfivedbysoflvfingtheffoflflowfingoptfimfizatfionprobflem:

maxX′

M′

m=1

1(x′∗am x′∗bm)+λK′

k′=1

(αk′+βk′) (23)

s.t. X′∗a,Θ=argmaxX′∗a,Θ

flogL(Θ;X′,X′∗a)

x′k′

mM′,K′

m,k′=1∈0,1,

whereX′=x′k′

mM′,K′

m,k′=1fistheattackstrategy,fi.e.,theflabefls

provfidedbythemaflficfiousworkersfforthefitemsfinO′.X′∗a=

x′∗amM′

m=1andX′∗b=x′∗bm

M′

m=1representtheestfimatedtrue

flabeflsffortheM′fitemsbasedonX′=X′∪X′andX′respectfivefly.

AflthoughtheattackstrategyX′derfivedbasedonEqn.(23)may

notbeasgoodasXbasedontheffuflflknowfledgeX,fitfistheoptfimaflchoficeffortheattackerfintheflfimfitedknowfledgescenarfio.TheperfformanceofftheproposedmechanfismwfithflfimfitedknowfledgefisevafluatedfinSectfion6.5.

6 EXPERIMENTSWeconductexperfimentsbasedonreafl-worfldcrowdsourcfingdatasetstoverfiffytheperfformanceofftheproposedfinteflflfigentattackmechanfism.

6.1 ExperfimentSetupInthfissectfion,wefintroducetheadoptedreafl-worfldcrowdsourc-fingdatasets,thebaseflfinemethodswhficharecomparedwfiththeproposedmechanfism,andtheperfformancemeasure.

6.1.1 Datasets.Toverfiffytheadvantagesofftheproposedfintefl-flfigentattackmechanfism,weadopttheffoflflowfingreafl-worfldcrowd-sourcfingdatasets.DuchenneSmfifleDataset[53].Inthfisdataset,thetaskfisto

judgewhetherthesfimfiflefinaffacefimage(anfitem)fisDuchenne(enjoymentsmfifle)orNon-Duchenne.Theauthorsfin[53]createtasksontheAmazonMechanficaflTurkpflatfform,andcoflflecttheflabeflsffromthepartficfipatfingworkers.Thenumberoffthefitemsfinthfisdatasetfis2,134.Totaflfly,thereare64normaflworkersandtheyprovfide17,729flabefls.ProductDataset[51,58].Eachfitemfinthfisdatasetcontafinstwo

products(wfithdescrfiptfions),thetaskfistojudgewhetherthetwoproductsarethesameornot.Thepartficfipatfingworkersneedtofidentfiffywhetherthetwodescrfiptfionsdescrfibethesameproductornot,andthenprovfidethefirflabefls.Inthfisdataset,thereare8,315fitemswhfichareobservedby176normaflworkers.Totaflfly,thesepartficfipatfingworkersprovfide24,945flabefls.SentfimentDataset[58].Eachfitemfinthedatasetfisatweet

reflatedtoacompany.Thepartficfipatfingworkersneedtofidentfiffywhetherthetweethasposfitfivesentfimentornottothecompany.Theauthorsfin[58


]create1,000fitemsandcoflflectflabeflsffrom85


18

normal workers through the AMT platform. Totally, there are 20,000labels in this dataset.

6.1.2 Baseline Methods. We compare the proposed attackmechanism with two baseline methods: Baseline_rand and Base-line_inversion.

In the Baseline_rand method, the attacker does not consider anystrategy, and he just randomly sets the labels of each maliciousworker on a given item. This method introduce less overhead to theattacker, as he does not need to take effort to obtain and analyzethe crowdsouring data collected from the normal workers.

In the Baseline_inversion method, the attacker first conducts theDawid-Skene model on the labels provided by the normal workersand get the estimated true label for each item. Then he sets eachmalicious worker’s label on a given item as the candidate answerwhich is different from the estimated true label. This method is anintuitive attack strategy, in which the attacker tries to maximizethe number of bad labels injected into the crowdsourcing data.

6.1.3 Performance Measure. In order to evaluate the perfor-mance of the proposed attack mechanism, we compare the aggrega-tion results before and after the data poisoning attacks, and adoptthe change rate as the measure metric. The change rate is definedas | |X

∗a−X ∗b | |M , where X ∗a = x∗am Mm=1 and X ∗b = x∗bm

Mm=1 are

the estimations for the items’ true labels after and before the datapoisoning attacks. Since the goal of the attacker is to maximize theerror of the aggregation results and meanwhile maximally raisethe reliability degrees of the malicious workers, thus, the larger thechange rate, the better the method.

6.2 The Effect of the Percentage of theMalicious Workers

When conducting the data poisoning attack, we assume that theattacker cannot manipulate the labels of normal workers, but hecan create or recruit multiple malicious workers. Thus, the numberof the malicious workers created or recruited by the attacker playsan important role in the attack. If the attacker is able to create orrecruit overwhelming number of malicious workers, the goal of theattacker can be easily achieved with the intuitive attack strategy, i.e.,the Baseline_inversion method. However, in practice, the attackercan only create or recruit a limited number of malicious workersdue to the limitation of his ability. In this experiment, we considerthe scenarios where the percentage of malicious workers is low,and evaluate the performance of the proposed mechanism whenthe percentage is varying.

Suppose N is the number of labels provided by the normal work-ers for all items. Here we assume that each malicious worker canobserve N /K items, which is the average number of the itemsobserved by each normal worker. For each malicious worker, theN /K observed items are randomly selected. In this paper, we setthe parameters θ and λ as 100 and 1, respectively. Then we varythe percentage of the malicious workers from 0.03 to 0.27. All theexperiments are conducted 50 times and we report the averageresults. The change rate for the three real-world crowdsourcingdatasets is shown in Figure 3, in which we represent the proposedmechanism as The intelligent attack. From this figure, we can seethe proposed attack mechanism performs better than the baseline

methods in all cases. When the percentage of the malicious workersis very low (e.g., 3%), since the malicious workers are too few tochange the final aggregation results much, the advantage of theproposed mechanism is small. However, when the percentage ofthe malicious workers increases, the advantage of the proposedattack scheme becomes bigger. For example, when the percentageof malicious workers is 27%, the proposed mechanism successfullyattacks nearly 50% of the items in the Duchenne Smile datasetswhile the baseline methods only obtain marginal utility.

0.03 0.09 0.15 0.21 0.27

Percentage of malicious workers

0

0.1

0.2

0.3

0.4

0.5

Ch

an

ge

rate

The intelligent attack

Baseline_rand

Baseline_inversion

(a)

0.03 0.09 0.15 0.21 0.27


0.00

0.05

0.10

0.15

0.20

0.25

0.30

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(b)

0.03 0.09 0.15 0.21 0.27


0.000

0.005

0.010

0.015

0.020

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(c)

Figure 3: Change rate w.r.t. the percentage of the maliciousworkers. (a): Duchenne Smile Dataset. (b): Product Dataset.(c): Sentiment Dataset.

6.3 The Effect of the Number of the QueriedItems

When the percentage of the malicious workers is given, the numberof the items queried to each malicious worker is another importantfactor in the attack. In this experiment, we study the performanceof the proposed mechanism when the number of the items queriedto each malicious worker varies.

Here we consider a scenario where the percentage of the ma-licious workers is very low and we set the value as 3%, i.e., theattacker creates or recruits 2, 6 and 3 malicious workers to thethree datasets, respectively. For the Duchenne Smile dataset andthe Product dataset, we vary the number of items queried to eachmalicious worker from 50 to 500, and for the Sentiment dataset, thenumber of the queried items varies from 100 to 700. The change ratefor the three datasets is shown in Figure 4. The results in this figureclearly verify that the proposed attack mechanism outperformsthe baseline methods in all cases. When the number of the itemsqueried to each malicious worker increases, the advantage of theproposed attack mechanism also increases. The reason is that withthe increment of the number of the queried items, the maliciousworkers can exert more impact on the final aggregation resultsbased on the proposed mechanism. Additionally, this figure alsoshows that the proposed mechanism can achieve good utility evenwith very few malicious workers. Take the Duchenne Smile datasetas an example, when each malicious worker provides 250 labels(less than the average number of that from normal workers), theproposed mechanism can successfully attack more than 10% of theitems with only 2 malicious workers.

6.4 Comparison on the Ability Parameters ofthe Malicious Workers

Besides maximizing the error of the aggregation results, the attackeralso tries to maximize the malicious workers’ reliability degrees (orability) such that they can be treated as high-quality workers andthus be disguised well. In fact, the proposed mechanism outper-forms the baseline methods mainly because we take the effect of the


19

100 200 300 400 500

Number of items

0.00

0.05

0.10

0.15

0.20

0.25

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(a)

100 200 300 400 500

Number of items

0.00

0.02

0.04

0.06

0.08

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(b)

100 200 300 400 500 600 700

Number of items

0.000

0.005

0.010

0.015

0.020

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(c)

Figure 4: Change rate w.r.t. the number of the items queriedto each malicious worker. (a): Duchenne Smile Dataset. (b):Product Dataset. (c): Sentiment Dataset.

malicious workers’ reliability degrees into account. The maliciousworkers can disguise themselves as good workers on some items toenhance their reliability degrees. For the baseline methods, sincethe malicious workers always disagree with the normal ones or ran-domly provide their labels, the attack behaviors may be detected bythe Dawid-Skene model and the malicious workers will be assignedwith low reliability degrees.

In this experiment, we investigate the distribution of the partic-ipating workers’ ability parameters, i.e., α = αk , αk ′

K,K ′k,k ′=1 and

β = βk , βk ′K,K ′k,k ′=1, which can be treated as the reliability degrees

of these workers based on the Dawid-Skene model. For each dataset,the percentage of themalicious workers is fixed as 5%.We report theresults of the parameters α and β for the three datasets after the datapoisoning attacks in Figure 5, Figure 6 and Figure 7, respectively.The results show that the malicious workers from the proposedmechanism have high reliability degrees (both α and β) comparingwith the normal workers. This means that the malicious workersblend into the normal workers successfully and they will be treatedas high-quality workers according to the Dawid-Skene model. Thisalso verifies that the proposed mechanism can well disguise themalicious behaviors of the attacker while maximizing the error ofthe aggregated results. In contrast, in the Baseline_inversionmethod,since the malicious workers always disagree with the normal work-ers, they will be assigned significantly low reliability degrees, whichnot only limit the performance of the malicious workers, but alsomake them easy to be detected. As for the Baseline_rand method,since the malicious workers randomly select their labels, the valuesof the ability parameters will be around 0.5. Although the maliciousworkers from the Baseline_rand method can disguise themselvesto some extent, their reliability degrees are not large enough toimpact the aggregated results.

6.5 The Effect of the Attacker’s KnowledgeAs described in Section 5, the proposed mechanism can also beemployed to implement an effective attack when the attacker onlyhas limited knowledge of the items’ normal labels. In this exper-iment, we evaluate the performance of the proposed mechanismwith respect to the value ofM ′/M , i.e., the percentage of the itemswhose labels from the normal workers can be known by the at-tacker. Here we still consider a scenario where the percentage ofthe malicious workers is very low (3%). We also assume that eachmalicious worker can observe N /K items, which are randomlyselected from O ′. Then we vary the value of M ′/M from 0.3 to1 and calculate the change rate for the three real-world datasets.We conduct the experiment for 50 times and report the average

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(a) The intelligent attack

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(b) Baseline_rand

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(c) Baseline_inversion

Figure 5: The ability parameters of the normal and mali-cious workers for the Duchenne Smile dataset.

20 50 80 110 140 170

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

20 50 80 110 140 170

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

20 50 80 110 140 170

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

20 50 80 110 140 170

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers


20 50 80 110 140 170

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(b) Baseline_rand

20 50 80 110 140 170

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers


Figure 6: The ability parameters of the normal and mali-cious workers for the Product dataset.

0 10 20 30 40 50 60 70 80 90

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

0 10 20 30 40 50 60 70 80 90

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

0 10 20 30 40 50 60 70 80 90

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

0 10 20 30 40 50 60 70 80 90

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers


0 10 20 30 40 50 60 70 80 90

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(b) Baseline_rand

0 10 20 30 40 50 60 70 80 90

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers


Figure 7: The ability parameters of the normal and mali-cious workers for the Sentiment dataset.

results in Figure 8, from which we can see the proposed mechanismoutperforms the baseline methods in all cases, and the advantageof the proposed mechanism becomes bigger when the attacker’sknowledge increases. These results verify that the proposed mech-anism can still achieve good utility when the attacker only haslimited knowledge of the items’ normal labels.

To further evaluate the performance of the proposed mechanismin the limited knowledge scenarios, we investigate the distributionof theworkers’ ability parameters when the attacker only has partial


20

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Percentage of knowledge

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(a)

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1


0.000

0.004

0.008

0.012

0.016

0.020

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(b)

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1


0.000

0.002

0.004

0.006

0.008

0.010

Ch

an

ge

rate


Baseline_rand

Baseline_inversion

(c)

Figure 8: Change rate w.r.t. the percentage of the knowl-edge known by the attacker (i.e.,M ′/M). (a): Duchenne SmileDataset. (b): Product Dataset. (c): Sentiment Dataset.

knowledge of the normal labels. Here we consider three cases inwhich the percentage of the known items (i.e.,M ′/M) is set as 0.3,0.5 and 0.7, respectively. In Figure 9 we report the results of theparameters α and β derived from the proposed mechanism on thethe Duchenne Smile Dataset. The results show that the maliciousworkers keep the high reliability degrees, which means that theproposed mechanism can well disguise the attack behaviors inthe limited knowledge scenarios. As for the baseline methods, theresults of them on the Duchenne Smile dataset are similar to thosein Figure 5.

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(a) M ′/M = 0.3

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(b) M ′/M = 0.5

5 15 25 35 45 55 65

Worker ID

0.0

0.2

0.4

0.6

0.8

1.0

Normal workers

Malicious workers

(c) M ′/M = 0.7

Figure 9: The workers’ ability parameters calculated by theintelligent attack mechanism in the limited knowledge sce-narios for the Duchenne Smile Dataset.

7 RELATED WORKAs an effective and low-cost way to solve challenging problems,crowdsourcing, which utilizes the wisdom of crowds, has becomemore and more popular [5, 10, 13, 15, 16, 27, 29, 31, 32, 36, 38–40].One important issue for crowdsourcing is that the participatingworkers are non-experts, so they are likely to provide noisy labels.To address this problem, the researchers have proposed many aggre-gation methods to estimate the true labels. Among these methods,the Dawid-Skene model [9] has been widely adopted in practice.Compared with the naive aggregationmethods such as majority vot-ing, the Dawid-Skene model takes the reliability degrees of workersinto account, and it can jointly estimate the items’ true labels andeach worker’s reliability degree. Although different variants havebeen developed and the theoretical analysis has been conductedfor the Dawid-Skene model [7, 8, 30, 34, 41, 44, 46, 57, 59], theseworks do not take into consideration the sophisticated data poi-soning attacks against this model in crowdsourcing. In this paper,we propose an effective data poisoning attack mechanism which

can maximize the error of the final results estimated based on theDawid-Skene model in crowdsourcing. With the spirit of disguisingthe malicious workers, the above methods cannot defend againstthis attack effectively.

There are also some crowdsourcing methods which are proposedto eliminate the spammers when collecting labels or conductingaggregation [12, 14, 18, 20, 22, 25, 43, 48]. However, the spammersdiscussed in these papers are usually the workers who uniformlyand/or randomly provide labels in crowdsourcing, which is similarto the Baseline_rand attack mechanism. As shown in various ex-periments, the proposed mechanism can launch significantly moreeffective attacks than the Baseline_rand. Since the malicious work-ers can disguise themselves well by providing labels intelligently,they will not be detected as spammers.

The importance of the data poisoning attacks has recently beenrecognized in many crowdsourcing and crowdsensing scenarios [6,17, 26, 42, 49, 50, 55]. Additionally, there also has been existing workthat investigates the data poisoning attacks and related defenseschemes in the applications of Internet of Things [21, 45, 56], electricpower grids [35] and machine learning algorithms [1, 3, 4, 19, 28,37, 54]. However, the attacked algorithms discussed in these papersare different from ours. In our designed mechanism, we study theoptimal data poisoning attacks against the crowdsourcing systemsempowered with the Dawid-Skene model. Compared with the naiveaggregation methods such as majority voting, the Dawid-Skenemodel can defend against the naive malicious workers to somedegree, which makes the attack more difficult. The most relevantpapers to this work are [23, 24], in which the proposed schemes canidentify the malicious workers who conduct the sophisticated datapoisoning attacks. However, based on these schemes, the workerswho agree with the majority will be classified as normal ones. Sincethe malicious workers in our proposed mechanism can disguisethemselves by agreeing with the majority on some items, thesemethods will fail to detect them.

8 CONCLUSIONSIn this paper, we investigate crowdsourcing in adversarial environ-ments and study the data poisoning attacks against the crowdsourc-ing systems with the Dawid-Skene model empowered. In order tofind an effective attack strategy for the attacker who aims to maxi-mize the error of the aggregated results, we design an intelligentattack mechanism, based on which an optimal attack strategy canbe derived by solving a bi-level optimization problem. With thederived optimal attack strategy, the attacker can not only achievemaximum attack utility but also intelligently disguise the intro-duced malicious workers as normal ones or even good ones. Theexperimental results based on real-world datasets demonstrate thatthe proposed attack mechanism can achieve higher attack utilitywith very few malicious workers and at the same time, is harder tobe detected by the defense mechanisms.

ACKNOWLEDGMENTSWe thank Dr. Niao He from the University of Illinois at Urbana-Champaign for her valuable suggestions. This work was supportedin part by the US National Science Foundation under grants CNS-1566374, CNS-1652503, IIS-1553411 and CNS-1742845.


21

REFERENCES[1] Scott Alfeld, Xiaojin Zhu, and Paul Barford. 2016. Data Poisoning Attacks against

Autoregressive Models. In Proc. of AAAI. 1452–1458.[2] Jonathan F Bard. 1998. Practical bilevel optimization: algorithms and applications.

Kluwer Academic Publishers.[3] Marco Barreno, Blaine Nelson, Russell Sears, Anthony D Joseph, and J Doug

Tygar. 2006. Can machine learning be secure?. In Proc. of ASIACCS. 16–25.[4] Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against

support vector machines. In Proc. of ICML.[5] Marco Brambilla, Stefano Ceri, Andrea Mauri, and Riccardo Volonterio. 2014.

Community-based crowdsourcing. In Proc. of WWW. 891–896.[6] Shih-Hao Chang and Zhi-Rong Chen. 2016. Protecting Mobile Crowd Sensing

against Sybil Attacks Using Cloud Based Trust Management System. MobileInformation Systems 2016 (2016).

[7] Xi Chen, Qihang Lin, and Dengyong Zhou. 2013. Optimistic knowledge gradientpolicy for optimal budget allocation in crowdsourcing. In Proc. of ICML. 64–72.

[8] Nilesh Dalvi, Anirban Dasgupta, Ravi Kumar, and Vibhor Rastogi. 2013. Aggre-gating crowdsourced binary ratings. In Proc. of WWW. 285–294.

[9] Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estima-tion of observer error-rates using the EM algorithm. Applied statistics (1979).

[10] Luca de Alfaro, Vassilis Polychronopoulos, andMichael Shavlovsky. 2015. Reliableaggregation of boolean crowdsourced tasks. In Proc. of HCOMP.

[11] Arthur P Dempster, NanM Laird, and Donald B Rubin. 1977. Maximum likelihoodfrom incomplete data via the EM algorithm. Journal of the royal statistical society(1977), 1–38.

[12] Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. 2012.Mechanical Cheat: Spamming Schemes and Adversarial Techniques on Crowd-sourcing Platforms.. In CrowdSearch. 26–30.

[13] Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. 2016.Scheduling human intelligence tasks in multi-tenant crowd-powered systems. InProc. of WWW. 855–865.

[14] Carsten Eickhoff and Arjen P de Vries. 2013. Increasing cheat robustness ofcrowdsourcing tasks. Information retrieval 16, 2 (2013), 121–137.

[15] Ju Fan, Guoliang Li, Beng Chin Ooi, Kian-lee Tan, and Jianhua Feng. 2015. icrowd:An adaptive crowdsourcing framework. In Proc. of SIGMOD. 1015–1030.

[16] Ujwal Gadiraju, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze. 2015.Human beyond the machine: Challenges and opportunities of microtask crowd-sourcing. IEEE Intelligent Systems 30, 4 (2015), 81–85.

[17] Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. 2015.Understandingmalicious behavior in crowdsourcing platforms: The case of onlinesurveys. In Proc. of CHI. 1631–1640.

[18] Matthias Hirth, Tobias Hoßfeld, and Phuoc Tran-Gia. 2010. Cheat-detectionmechanisms for crowdsourcing. University of Würzburg, Tech. Rep 4 (2010).

[19] Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and JDTygar. 2011. Adversarial machine learning. In Proc. of AISec. 43–58.

[20] Nguyen Quoc Viet Hung, Duong Chi Thang, Matthias Weidlich, and Karl Aberer.2015. Minimizing efforts in validating crowd answers. In Proc. of SIGMOD. 999–1014.

[21] Vittorio P Illiano and Emil C Lupu. 2015. Detecting malicious data injections inwireless sensor networks: A survey. ACM Computing Surveys (CSUR) (2015).

[22] Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. 2010. Quality managementon amazon mechanical turk. In Proc. of the ACM SIGKDD workshop on humancomputation. 64–67.

[23] Srikanth Jagabathula, Lakshminarayanan Subramanian, and Ashwin Venkatara-man. 2014. Reputation-based worker filtering in crowdsourcing. In Proc. of NIPS.2492–2500.

[24] Srikanth Jagabathula, Lakshminarayanan Subramanian, and Ashwin Venkatara-man. 2016. Identifying Unreliable and Adversarial Workers in CrowdsourcedLabeling Tasks. (2016).

[25] David R Karger, Sewoong Oh, and Devavrat Shah. 2014. Budget-optimal taskallocation for reliable crowdsourcing systems. Operations Research 62, 1 (2014),1–24.

[26] Walter S Lasecki, Jaime Teevan, and Ece Kamar. 2014. Information extractionand manipulation threats in crowd-powered systems. In Proc. of CSCW. 248–256.

[27] Edith Law, Ming Yin, Joslin Goh, Kevin Chen, Michael A Terry, and Krzysztof ZGajos. 2016. Curiosity killed the cat, but makes crowdwork better. In Proc. of CHI.4098–4110.

[28] Bo Li, YiningWang, Aarti Singh, and Yevgeniy Vorobeychik. 2016. Data poisoningattacks on factorization-based collaborative filtering. In Proc. of NIPS. 1885–1893.

[29] Guoliang Li, Jiannan Wang, Yudian Zheng, and Michael J Franklin. 2016. Crowd-sourced data management: A survey. IEEE Transactions on Knowledge and DataEngineering 28, 9 (2016), 2296–2319.

[30] Hongwei Li, Bin Yu, and Dengyong Zhou. 2013. Error rate analysis of labelingby crowdsourcing. In ICML Workshop: Machine Learning Meets Crowdsourcing.

[31] Qi Li, Fenglong Ma, Jing Gao, Lu Su, and Christopher J Quinn. 2016. Crowdsourc-ing high quality labels with a tight budget. In Proc. of WSDM. 237–246.

[32] Yaliang Li, Jing Gao, Patrick PC Lee, Lu Su, Caifeng He, Cheng He, Fan Yang,and Wei Fan. 2017. A weighted crowdsourcing approach for network quality

measurement in cellular data networks. IEEE Transactions on Mobile Computing16, 2 (2017), 300–313.

[33] Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures onhuman language technologies 5, 1 (2012), 1–167.

[34] Qiang Liu, Jian Peng, and Alexander T Ihler. 2012. Variational inference forcrowdsourcing. In Proc. of NIPS. 692–700.

[35] Yao Liu, Peng Ning, and Michael K Reiter. 2011. False data injection attacksagainst state estimation in electric power grids. ACM Transactions on Informationand System Security 14, 1 (2011), 13.

[36] Fenglong Ma, Yaliang Li, Qi Li, Minghui Qiu, Jing Gao, Shi Zhi, Lu Su, Bo Zhao,Heng Ji, and Jiawei Han. 2015. Faitcrowd: Fine grained truth discovery forcrowdsourced data aggregation. In Proc. of KDD. 745–754.

[37] Shike Mei and Xiaojin Zhu. 2015. Using Machine Teaching to Identify OptimalTraining-Set Attacks on Machine Learners. In Proc. of AAAI. 2871–2877.

[38] Chuishi Meng, Wenjun Jiang, Yaliang Li, Jing Gao, Lu Su, Hu Ding, and YunCheng. 2015. Truth discovery on crowd sensing of correlated entities. In Proc. ofSenSys. 169–182.

[39] Chenglin Miao, Wenjun Jiang, Lu Su, Yaliang Li, Suxin Guo, Zhan Qin, HoupingXiao, Jing Gao, and Kui Ren. 2015. Cloud-enabled privacy-preserving truthdiscovery in crowd sensing systems. In Proc. of SenSys. 183–196.

[40] Quoc Viet Hung Nguyen, Tam Nguyen Thanh, Ngoc Tran Lam, Son Thanh Do,and Karl Aberer. 2013. A Benchmark for Aggregation Techniques in Crowdsourc-ing. In Proc. of SIGIR.

[41] Jungseul Ok, Sewoong Oh, Jinwoo Shin, and Yung Yi. 2016. Optimality of beliefpropagation for crowdsourced classification. In Proc. of ICML. 535–544.

[42] Zhengrui Qin, Qun Li, and George Hsieh. 2013. Defending against coopera-tive attacks in cooperative spectrum sensing. IEEE Transactions on WirelessCommunications 12, 6 (2013), 2680–2687.

[43] Vikas C Raykar and Shipeng Yu. 2012. Eliminating spammers and rankingannotators for crowdsourced labeling tasks. Journal of Machine Learning Research13, Feb (2012), 491–518.

[44] Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, CharlesFlorin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. Journal ofMachine Learning Research 11, Apr (2010), 1297–1322.

[45] Mohsen Rezvani, Aleksandar Ignjatovic, Elisa Bertino, and Sanjay Jha. 2015.Secure data aggregation technique for wireless sensor networks in the presenceof collusion attacks. IEEE Transactions on Dependable and Secure Computing 12, 1(2015), 98–110.

[46] Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheapand fast—but is it good?: evaluating non-expert annotations for natural languagetasks. In Proc. of the EMNLP. 254–263.

[47] Norases Vesdapunt, Kedar Bellare, and Nilesh Dalvi. 2014. Crowdsourcing algo-rithms for entity resolution. Proceedings of the VLDB Endowment 7, 12 (2014),1071–1082.

[48] Jeroen Vuurens, Arjen P de Vries, and Carsten Eickhoff. 2011. How much spamcan you take? an analysis of crowdsourcing results to increase accuracy. In Proc.of CIR. 21–26.

[49] Gang Wang, Bolun Wang, Tianyi Wang, Ana Nika, Haitao Zheng, and Ben YZhao. 2016. Defending against sybil devices in crowdsourced mapping services.In Proc. of MobiSys. 179–191.

[50] GangWang, TianyiWang, Haitao Zheng, and Ben Y Zhao. 2014. Man vs. Machine:Practical Adversarial Detection of Malicious CrowdsourcingWorkers.. In USENIXSecurity Symposium. 239–254.

[51] Jiannan Wang, Tim Kraska, Michael J Franklin, and Jianhua Feng. 2012. Crowder:Crowdsourcing entity resolution. Proc. of the VLDB Endowment 5, 11 (2012),1483–1494.

[52] Peter Welinder and Pietro Perona. 2010. Online crowdsourcing: rating annotatorsand obtaining cost-effective labels. In Proc. of CVPRW. 25–32.

[53] Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul LRuvolo. 2009. Whose vote should count more: Optimal integration of labels fromlabelers of unknown expertise. In Proc. of NIPS. 2035–2043.

[54] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, andFabio Roli. 2015. Is feature selection secure against training data poisoning?. InProc. of ICML. 1689–1698.

[55] Dong Yuan, Guoliang Li, Qi Li, and Yudian Zheng. 2017. Sybil Defense in Crowd-sourcing Platforms. In Proc. of CIKM. 1529–1538.

[56] Kuan Zhang, Xiaohui Liang, Rongxing Lu, and Xuemin Shen. 2014. Sybil attacksand their defenses in the internet of things. IEEE Internet of Things Journal 1, 5(2014), 372–383.

[57] Yuchen Zhang, Xi Chen, Denny Zhou, and Michael I Jordan. 2014. Spectralmethods meet EM: A provably optimal algorithm for crowdsourcing. In Proc. ofNIPS. 1260–1268.

[58] Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng. 2017.Truth inference in crowdsourcing: is the problem solved? Proc. of the VLDBEndowment 10, 5 (2017), 541–552.

[59] Denny Zhou, Sumit Basu, Yi Mao, and John C Platt. 2012. Learning from thewisdom of crowds by minimax entropy. In Proc. of NIPS. 2195–2203.


22

Documents

Attack under Disguise: An Intelligent Data Poisoning ...mh6ck/papers/ · Amazon Mechanical Turk (AMT)1 and CrowdFlower2, crowdsourc-ing has emerged as a popular, fast and cheap problem-solving