Embed Size (px)
Citation preview
MASSIF: Interactive Interpretation ofAdversarial Attacks on Deep Learning
Nilaksh Das*Georgia Institute of TechnologyAtlanta, GA, [email protected]
Robert FirstmanGeorgia Institute of TechnologyAtlanta, GA, [email protected]
Haekyu Park*Georgia Institute of TechnologyAtlanta, GA, [email protected]
Emily RogersGeorgia Institute of TechnologyAtlanta, GA, [email protected]
Zijie J. WangGeorgia Institute of TechnologyAtlanta, GA, [email protected]
Duen Horng (Polo) ChauGeorgia Institute of TechnologyAtlanta, GA, [email protected]
Fred HohmanGeorgia Institute of TechnologyAtlanta, GA, [email protected]
*Authors contributed equally.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).CHI ’20 Extended Abstracts, April 25–30, 2020, Honolulu, HI, USA.© 2020 Copyright is held by the author/owner(s).ACM ISBN 978-1-4503-6819-3/20/04.http://dx.doi.org/10.1145/3334480.3382977
AbstractDeep neural networks (DNNs) are increasingly poweringhigh-stakes applications such as autonomous cars andhealthcare; however, DNNs are often treated as “blackboxes” in such applications. Recent research has also re-vealed that DNNs are highly vulnerable to adversarial at-tacks, raising serious concerns over deploying DNNs in thereal world. To overcome these deficiencies, we are develop-ing MASSIF, an interactive tool for deciphering adversarialattacks. MASSIF identifies and interactively visualizes neu-rons and their connections inside a DNN that are stronglyactivated or suppressed by an adversarial attack. MASSIF
provides both a high-level, interpretable overview of theeffect of an attack on a DNN, and a low-level, detailed de-scription of the affected neurons. MASSIF’s tightly coupledviews help people better understand which input featuresare most vulnerable and important for correct predictions.
Author KeywordsDeep learning interpretability, adversarial attack, visual ana-lytics, scalable summarization, attribution graph
IntroductionDeep neural networks (DNNs) have demonstrated signifi-cant success in a wide spectrum of applications [3, 5, 6, 9].However, they have been found to be highly vulnerable toadversarial attacks: typically small, human-imperceptible
perturbations on inputs that fool DNNs into making incor-rect predictions [2, 4, 8, 11]. This jeopardizes many DNN-based technologies, especially in security and safety-criticalapplications such as autonomous driving and data-drivenhealthcare. To make deep learning more robust againstsuch malicious attacks, it is essential to understand how theattacks permeate DNN models [12, 14]. Interpreting, andultimately defending against adversarial attacks, is nontrivialdue to a number of challenges:
C1. Scaling to large datasets. Since DNNs contain thou-sands of neurons and connections, summarizing howan attack affects a model requires analyzing the model’sbehavior over entire datasets and developing scalablerepresentations that characterize the attack [1, 7].
C2. Entangled features and connections between be-nign and attacked inputs. A natural approach forunderstanding adversarial attacks is to compare amodel’s operations on benign and attacked inputs,which could help people understand where and whypredictions within a model diverge. However, designingan effective comparison can be challenging becausethe features contributing to the differences may cor-relate with one or more features in both the benignand attacked classes. For example, Figure 1 showsthat a feature representing “ivory face with dark eyesand nose” (center purple node) is important for boththe benign panda class and the attacked armadilloclass. This shared feature correlates with a feature forpanda (e.g., “black & white patches”), while also corre-lating with a few features for armadillo (e.g., “scales”,“crossed pattern”).
C3. Diverse feature vulnerability. Given an adversarialattack, some learned features may be more easily ma-nipulated and vulnerable than others. For example,
PandaBenign
ArmadilloTarget
= +Panda Attack
Highly activated inBenign
Highly activated inAttacked
Ivory face,dark eyes, noseIvory face,
dark eyes, nose
Black & whitepatches
Black & whitepatches
CrossedpatternCrossedpattern
ScalesScales
Figure 1: Adversarial attacks confuse DNNs to make incorrectpredictions, e.g., attacking benign panda images so they aremisclassified as armadillo. We aim to understand where suchattacks occur inside the model and what features are used.
manipulating a “basketball” feature into an “orange”feature is easier (similar shape and color) than chang-ing it into a “truck” feature. As features exhibit a spec-trum of vulnerability, enabling users to visualize andunderstand an attack under varying levels of severitycould help them design stronger countermeasures.
ContributionsTo address the aforementioned challenges, we are develop-ing MASSIF, an interactive visualization tool for interpretingadversarial attacks on deep learning models. Our ongoingwork presents the following contributions:
1. Novel graph-based comparison. To discover thefeatures and connections activated or suppressed byan attack, we adapt the recently proposed attribution
graph [7] in a novel way to visualize, summarize, andcompare a model’s response to benign and attackeddata. The original attribution graph aims to highlighthow a model’s learned features interact to make predic-tions for a single class, by representing highly activatedneurons as vertices and their most influential connec-tions as edges. Our main idea for MASSIF is to generateand integrate two attribution graphs: one for the benigndata and another for the attacked data, as illustrated inFigure 1. The aggregated graph helps us understandwhich features are shared by both benign and attackeddata (e.g., purple, center feature in Figure 1), which aresolely activated by the benign data (blue, far left), andwhich are by the attacked data (red, far right). Impor-tantly, MASSIF also helps users more easily discoverwhere a prediction starts to “diverge”, honing in to thecritical parts of the model that the attack is exploiting.
2. Fractionation of neurons based on vulnerability. Tohelp users prioritize their inspection of neurons, we de-velop a new way to sort and group them based on theirvulnerability, i.e., “how easily can a neuron be activatedor suppressed by an attack.” Our main idea is to vary anattack’s strength (or severity) and record all neuron acti-vations. Neurons that are easily activated or suppressedby solely a weak attack may warrant focused inspectionsince they can be easily manipulated with little effort.
Example patches
Scales pattern
Example patches Example patchesBumpy texture
3
Interface
ControlPanel
A
AttributionGraph View
B
C
D
B1B2
B3Shared features in benign and attacked class
Features added by the attack (closer to Armadillo)
Example patchesExample patches
Example patches
128
1
32
787 135747 734
4e
5b
5a
4d
4c
FeatureVisualization
ExamplePatches
InfluentialPaths
Example patches
Example patches Example patches Example patches Example patches
Example patches
FeatureVisualization
Figure 2: Each neuron in anattribution graph is representedwith its feature visualization. Whena user hovers over a neuron,example dataset patches areshown for context.
System Design and ImplementationThis section describes MASSIF’s interface and visualizationdesign (see Figure 3). Users can select a benign class, anattacked class, and the type and severity of an attack fromthe top control panel (Figure 3A). Then, MASSIF generatesattribution graphs using a Python backend, and displaysthem to the user in a web-based interface built with HTML,CSS, JavaScript, and D3 (Figure 3B).
Nodes: Features activated (or suppressed) by attacks.In attribution graphs, nodes represent DNN neurons whichare trained to detect particular features in input data. To in-terpret what features a neuron detects, MASSIF representseach neuron with its feature visualization: a synthesized im-age that maximizes the neuron’s activation [10]. Users canhover over any neuron’s feature visualization to also displayexample image patches from the dataset that most activatethat neuron. For example, as seen in Figure 2, the featurevisualization (left) and dataset examples (right) describe aneuron that detects a dotted pattern in scales.
We divide an attribution graph’s neurons into three groupsby their attack response. First, suppressed neurons arehighly activated by benign inputs but become suppressedby adversarial inputs. These represent crucial features forthe benign class, but the model fails to detect them whenexposed to the attack. Second, emphasized neurons arenot noticeably activated by benign inputs but become highlyactivated by adversarial inputs. These represent featuresthat are typically not important for the benign class, but themodel detects them as important features of the attackedclass. Third, shared neurons are highly activated by andimportant to both benign and adversarial inputs.
We visually distinguish these three neuron groups with dif-ferent colors and positions in the attribution graph view (Fig-ure 3B). Suppressed neurons are colored blue and posi-tioned on the left (Figure 3B.1). Emphasized neurons arecolored orange and positioned on the right (Figure 3B.3).Shared neurons are colored purple and positioned in themiddle between suppressed neurons and emphasized neu-rons (Figure 3B.2). The result is a visualization that disen-tangles and compares the DNN features and connectionsfrom the benign and attacked data.
Example patches
More vulnerable to the attack
Scales patternExample patches
BasketsExample patches
Bumpy texture
ControlPanel
A
AttributionGraph View
B
C
DB1B2
B3Shared features in benign and attacked class
4e
5b
5a
4d
4c
FeatureVisualization
ExamplePatches
InfluentialPaths
128Example patches
1 32
Figure 3: The MASSIF interface. A user Hailey is studying the targeted Fast Gradient Method (FGM) attack performed on theInceptionV1 model. Using the control panel (A), she selects “giant panda” as the benign class and “armadillo” as the attack tar-get class. MASSIF generates an attribution graph (B), which shows Hailey the neurons within the network that are suppressed inthe attacked images (B1, blue), shared by both benign and attacked images (B2, purple), and emphasized only in the attackedimages (B3, orange). Each neuron is represented by a node and its feature visualization (C). Hovering over any neuron dis-plays example dataset patches that maximally activate the neuron, providing stronger evidence for what a neuron has learnedto detect. Hovering over a neuron also highlights its most influential connections from the previous layer (D), allowing Hailey todetermine where in the network the prediction diverges from the benign class to the attacked class.
Fractionation: Characterizing neuron vulnerability.Within each group of neurons, we further distinguish thembased on their vulnerability. A neuron is considered morevulnerable if its activation changes greatly under weaker at-tacks. We encode neuron vulnerability using its position andcolor within its group. More vulnerable neurons are locatedcloser to shared neurons, since they are on the border ofthe benign and attacked classes, and cause misclassifica-tion under weaker attacks. Suppressed neurons that arecloser to the border with shared neurons are colored pur-ple/blue, and emphasized neurons that are closer to theborder with shared neurons are colored purple/orange.
734787747
128
135
Example patches
Scales pattern
Example patches Example patchesBumpy texture
3
128
787 135747 734
InfluentialPaths
Example patches
Example patches Example patches Example patches Example patches
128
787 135747 734
Attacked PandasAInfluential
Paths
Interface
ControlPanel
A
AttributionGraph View
B
C
D
B1B2
B3Shared features in benign and attacked class
Features added by the attack (closer to Armadillo)
Example patchesExample patches
Example patches
1
32
4e
5b
5a
4d
4c
FeatureVisualization
ExamplePatches
Example patches
FeatureVisualization
Benign PandasB
Figure 4: Edges in an attributiongraph represent influentialconnections between neurons.(A) above shows a part of anattribution graph from Figure 3D.An emphasized neuron isconnected to one shared neuron(purple) and three emphasizedneurons (orange) from theprevious layer. (B) shows how thesame part of the attribution graphlooks different for benign images.The three emphasized neurons inthe previous layer are not activatedby the benign inputs, which causesthe emphasized neuron in thecurrent layer to be less activated.
Edges: Explaining why features are activated (or sup-pressed). In attribution graphs, edges represent influen-tial connections between neurons that most interact witheach other to represent a particular class [7]. These con-nections can explain why an adversarial feature is detectedand why some features are suppressed using attribution.
With MASSIF, users can interactively visualize attribution,i.e., drilling down into specific subgraphs by hovering overa neuron and highlighting its previous connections whichmaximally influence the neuron’s activation (Figure 3D).When inspecting a particular emphasized neuron n, thehighlighted connected neurons from the previous layer willbe either shared neurons or other emphasized neurons, asseen in Figure 4A. Users can observe that the connectedemphasized neurons from the previous layer contributehighly to the activation of neuron n. Similarly, to understandwhy a feature becomes suppressed by an attack, inspectinga particular suppressed neuron m highlights its connectedneurons from the previous layer. These will be either sharedneurons or suppressed neurons. The neuron m is less ac-tivated since the influential neurons from the previous layerare no longer activated, i.e., they are suppressed as well.
Preliminary ResultsWe present usage scenarios showing how MASSIF canhelp users better understand adversarial attacks on deeplearning models. Our user Hailey is studying a targetedversion of Fast Gradient Method [4] applied on the Incep-tionV1 model [13]. The model is trained on the ImageNetdataset, which contains over 1.2 million images across1,000 classes. Using the control panel (Figure 3A), sheselects “giant panda" as the benign class and “armadillo" asthe target class. She sets the maximum attack strength 3.5.
Which neurons are attacked? Hailey starts by findingwhich specific neurons are attacked to narrow down thepart of the model to investigate. In the attribution graphview (Figure 3B), she hovers over the suppressed neurons(Figure 3B.1) and the emphasized neurons (Figure 3B.3).She sees which features are emphasized and suppressedusing the neuron feature visualization and dataset examplepatches. Exploring these features, she finds the mixed5alayer interesting because three emphasized neurons (223,698, and 128) in mixed5a look related to armadillo skinpatterns (Figure 3C). She decides to focus on these em-phasized neurons in mixed5a.
Which neurons are easily attacked? To efficiently de-vise a countering defense, Hailey wants to prioritize theneurons and investigate them in order. She knows thatMASSIF fractionates the neurons according to how easilythey are attacked; therefore, she checks how the empha-sized neurons in mixed5a are separated (Figure 3C). Byhovering over the neurons from left to right, she observesthat “scales pattern” is most vulnerable, followed by “bas-kets” and “bumpy texture” neurons. She decides to explorethe neurons in this order, since she presumes that it is moreefficient to protect more vulnerable neurons.
Why are these neurons attacked? Hailey now wantsto know how to protect the attacked neurons related to ar-madillo skin patterns. She sequentially observes the attri-bution for the “scales pattern”, “baskets”, and “bumpy tex-ture” neurons. Upon inspecting the “bumpy texture” neuron(Figure 3D), MASSIF shows that four neurons in the pre-vious layer are highly interacting with it: a shared neuronrepresenting “black circle,” and emphasized neurons repre-senting “spider legs”, “granular texture”, and “a white hairydog’s face.” As the three emphasized neurons in the previ-ous layer can be the primary reason behind the detection of“bumpy texture”, she decides to investigate these neuronsmore using MASSIF.
Ongoing WorkInteractive neuron editing. MASSIF currently visualizesthe neurons that are activated or suppressed by an attackunder varying degrees of severity. We are working on ex-tending MASSIF’s interactivity by allowing real-time neuronediting, e.g., deletion. This would allow a user to activelyidentify vulnerable neurons using our visualization and in-teractively remove them from the DNN to observe its effectin real-time. Neuron deletion would mask the activationsof a particular neuron, potentially preventing the maliciouseffect of a targeted attack to propagate deeper into the net-work. This would enable a user to preemptively edit a DNNto enhance its robustness to adversarial attacks. For ex-ample, a user may identify and choose to delete a sharedneuron that only feeds into emphasized neurons, prevent-ing adversarially activated neurons from having any effectin the subsequent layers of the network, thus thwarting thetargeted attack from succeeding.
Planned evaluation. We plan to evaluate the effective-ness of our visualization tool coupled with interactive neu-ron editing through in-lab user studies where participants
seek to increase the robustness of a large-scale, pretrainedDNN model. We will recruit students with basic knowledgeof deep learning models. All participants will be asked editthe DNN for different benign-attacked class pairs and willbe evaluated on the basis of reduction in targeted attacksuccess rate. We will also conduct pre-test and post-testsurveys to evaluate whether MASSIF gave any deeper in-sights into the failure modes of the studied DNN and whatfactors the participants considered while editing the DNN toincrease its robustness to adversarial attacks.
ConclusionWe present MASSIF, an interactive system we are develop-ing that visualizes how adversarial attacks permeate DNNmodels and cause misclassification. MASSIF generates andvisualizes multiple attribution graphs as a summary of whatfeatures are important for a particular class (e.g., benign orattacked class) and how the features are related. MASSIF
enables flexible comparison between benign and attackedattribution graphs, highlighting where and why the attribu-tion graphs start to diverge, ultimately helping people betterinterpret complex deep learning models, their vulnerabili-ties, and how to best construct defenses.
AcknowledgementsThis work was supported in part by NSF grants IIS-1563816,CNS-1704701, NASA NSTRF, DARPA GARD, gifts from In-tel (ISTC-ARSA), NVIDIA, Google, Symantec, Yahoo! Labs,eBay, Amazon.
REFERENCES[1] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian
Johnson, and Chris Olah. 2019. Activation atlas. Distill4, 3 (2019), e15.
[2] Shang-Tse Chen, Cory Cornelius, Jason Martin, andDuen Horng Polo Chau. 2018. Shapeshifter: Robust
physical adversarial attack on faster r-cnn objectdetector. In Joint European Conference on MachineLearning and Knowledge Discovery in Databases.Springer, 52–68.
[3] Andre Esteva, Alexandre Robicquet, BharathRamsundar, Volodymyr Kuleshov, Mark DePristo,Katherine Chou, Claire Cui, Greg Corrado, SebastianThrun, and Jeff Dean. 2019. A guide to deep learningin healthcare. Nature medicine 25, 1 (2019), 24.
[4] Ian J. Goodfellow, Jonathon Shlens, and ChristianSzegedy. 2014. Explaining and HarnessingAdversarial Examples. CoRR abs/1412.6572 (2014).
[5] Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias,and Gigel Macesanu. 2019. A survey of deep learningtechniques for autonomous driving. Journal of FieldRobotics (2019).
[6] Guodong Guo and Na Zhang. 2019. A survey on deeplearning based face recognition. Computer Vision andImage Understanding 189 (2019), 102805.
[7] Fred Hohman, Haekyu Park, Caleb Robinson, andDuen Horng Chau. 2019. Summit: Scaling DeepLearning Interpretability by Visualizing Activation andAttribution Summarizations. IEEE VIS (2019).
[8] Alexey Kurakin, Ian Goodfellow, and Samy Bengio.2016. Adversarial examples in the physical world.arXiv preprint arXiv:1607.02533 (2016).
[9] Ali Bou Nassif, Ismail Shahin, Imtinan Attili,Mohammad Azzeh, and Khaled Shaalan. 2019.
Speech recognition using deep neural networks: Asystematic review. IEEE Access 7 (2019),19143–19165.
[10] Chris Olah, Alexander Mordvintsev, and LudwigSchubert. 2017. Feature Visualization. Distill (2017).DOI:http://dx.doi.org/10.23915/distill.00007https://distill.pub/2017/feature-visualization.
[11] Yao Qin, Nicholas Carlini, Ian Goodfellow, GarrisonCottrell, and Colin Raffel. 2019. Imperceptible, robust,and targeted adversarial examples for automaticspeech recognition. arXiv preprint arXiv:1903.10346(2019).
[12] Andrew Slavin Ross and Finale Doshi-Velez. 2018.Improving the adversarial robustness andinterpretability of deep neural networks by regularizingtheir input gradients. In Thirty-second AAAI conferenceon artificial intelligence.
[13] Christian Szegedy, Wei Liu, Yangqing Jia, PierreSermanet, Scott Reed, Dragomir Anguelov, DumitruErhan, Vincent Vanhoucke, and Andrew Rabinovich.2015. Going deeper with convolutions. In Proceedingsof the IEEE conference on computer vision and patternrecognition. 1–9.
[14] Guanhong Tao, Shiqing Ma, Yingqi Liu, and XiangyuZhang. 2018. Attacks meet interpretability:Attribute-steered detection of adversarial samples. InAdvances in Neural Information Processing Systems.7717–7728.
