CMU Usable Privacy and Security Laboratoryhttp://cups.cs.cmu.edu/

Protecting People from Phishing: The Design and

Evaluation of an Embedded Training Email System

P. Kumaraguru, Y. Rhee, A. Acquisti, L. Cranor, J. Hong, E. Nunge

Phishing emailPhishing email

Phishing emailPhishing email

Subject: eBay: Urgent Notification From Billing Department

Phishing emailPhishing email

We regret to inform you that you eBay account could be suspended if you don’t update your account information.

Phishing emailPhishing email

https://signin.ebay.com/ws/eBayISAPI.dll?SignIn&sid=verify&co_partnerid=2&sidteid=0

Phishing websitePhishing website

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 7

What is phishing?What is phishing?

Phishing is “a broadly launched social engineering attack in which an electronic identity is misrepresented in an attempt to trick individuals into revealing personal credentials that can be used fraudulently against them.”

Financial Services Technology Consortium. Understanding and countering the phishing threat: A financial serviceindustry perspective. 2005.

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 8

Phishing is growingPhishing is growing 73 million US adults received more than 50

phishing emails a year in 2005

Gartner found approx. 30% users changed online banking behavior because of attacks like phishing in 2006

Gartner predicted $2.8 billion loss in 2006

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 9

Why phishing is a hard problem?Why phishing is a hard problem? Semantic attacks take advantage of the

way humans interact with computers

Phishing is one type of semantic attack

Phishers make use of the trust that users have on legitimate organizations

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 10

Counter measures for phishingCounter measures for phishing Silently eliminating the threat

• Regulatory & policy solutions

• Email filtering (SpamAssasin)

Warning users about the threat• Toolbars (SpoofGuard, TrustBar)

Training users not to fall for attacks

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 11

Why user education is hard?Why user education is hard? Security is a secondary task (Whitten et al.)

Users are not motivated to read privacy policies (Anton et al.)

Reading existing online training materials creates concern among users (Anandpara et al.)

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 12

Our hypothesesOur hypotheses Security notices are an ineffective medium

for training users

Users make better decision when trained by embedded methodology compared to security notices

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 13

Design constraintsDesign constraints People don’t proactively read the training

materials on the web

Organizations send “security notices” to train users and people don’t read security notices

People can learn from web-based training materials, if only we could get people to read them! (Kumaraguru, 2006)

P. Kumaraguru, S. Sheng, A. Acquisti, L. Cranor, and J. Hong. Teaching Johnny Not to Fall for Phish.

Tech. rep., Cranegie Mellon University, 2007. http://www.cylab.cmu.edu/files/cmucylab07003.pdf.

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 14

Embedded training Embedded training We know people fall for phishing emails

So make training available through the phishing emails

Training materials are presented when the users actually fall for phishing emails

Embedded training exampleEmbedded training example

Subject: Revision to Your Amazon.com Information

Embedded training exampleEmbedded training example

Subject: Revision to Your Amazon.com Information

Please login and enter your information

http://www.amazon.com/exec/obidos/sign-in.html

Comic strip interventionComic strip intervention

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 18

Design rationaleDesign rationale What to show in the intervention?

When to show the intervention?

Analyzed instructions from most popular websites

Paper and HTML prototypes, 7 users each

Lessons learned • Two designs

• Present the training materials when users click on the link

Comic strip interventionComic strip intervention

Intervention #1 - Comic strip

Intervention #1 - Comic strip

Intervention #1 - Comic strip

Intervention #2 - Graphics and textIntervention #2 - Graphics and text

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 24

Study designStudy design

Think aloud study

Role play as Bobby Smith, 19 emails including 2 interventions, and 4 phishing emails

Three conditions: security notices, text / graphics intervention, comic strip intervention

10 non-expert participants in each condition, 30 total

Intervention #1 - Security noticesIntervention #1 - Security notices

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 26

Intervention #2 - Graphics and textIntervention #2 - Graphics and text

Intervention #3 - Comic stripIntervention #3 - Comic strip

Phish TrainingLegitimate Spam

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 29

User study - resultsUser study - results We treated clicking on link to be falling for

phishing

93% of the users who clicked went ahead and gave personal information

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 30

User study - resultsUser study - results

0

10

20

30

40

50

60

70

80

90

100

3: Phish 5:Training

7: Legit 11:Training

13: Legit 14:Phish-N

16:Phish-N

17:Phish

Emails which had a link in them

Pe

rce

nta

ge

of

use

rs w

ho

clic

ked

on

th

e li

nk

Notices Text / Graphics Comic

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 31

User study - resultsUser study - results Significant difference between security

notices and the comic strip group (p-value < 0.05)

Significant difference between the comic and the text / graphics group (p-value < 0.05)

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 32

ConclusionConclusion H1: Security notices are an ineffective

medium for training usersSup

porte

d

H2: Users make better decision when trained by embedded methodology compared to security notices Sup

porte

d

Latest comic strip designLatest comic strip design

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 34

Ongoing workOngoing work Measuring knowledge retention and

knowledge transfer• Knowledge retention is the ability to apply the

knowledge gained from one situation to another same or similar situation after a time period

• Knowledge transfer is the ability to transfer the knowledge gained from one situation to another situation after a time period

Is falling for phishing necessary for training?

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 35

Coming upComing up WWW 2007

• CANTINA: A Content-Based Approach to Detecting Phishing Web Sites

• Learning to Detect Phishing Emails

Our other research in anti-phishing http://cups.cs.cmu.edu/trust.php

Symposium On Usable Privacy and Security (SOUPS), July 18 - 20, 2007 at Carnegie Mellon University

• CMU Usable Privacy and Security Laboratory • http://www.cs.cmu.edu/~ponguru 36

AcknowledgementsAcknowledgements

Members of Supporting Trust Decision research group

Members of CUPS lab

CMU Usable Privacy and Security Laboratory

http://cups.cs.cmu.edu/