Alice in Warningland:A Large-Scale Field Study of Browser Security Warning Effectiveness

Devdatta AkhaweUniversity of California, Berkeley\ast

devdatta@cs.berkeley.edu

Adrienne Porter FeltGoogle, Inc.

felt@google.com

AbstractWe empirically assess whether browser security warn-

ings are as ineffective as suggested by popular opinionand previous literature. We used Mozilla Firefox andGoogle Chromes in-browser telemetry to observe over25 million warning impressions in situ. During our fieldstudy, users continued through a tenth of Mozilla Fire-foxs malware and phishing warnings, a quarter of GoogleChromes malware and phishing warnings, and a third ofMozilla Firefoxs SSL warnings. This demonstrates thatsecurity warnings can be effective in practice; securityexperts and system architects should not dismiss the goalof communicating security information to end users. Wealso find that user behavior varies across warnings. In con-trast to the other warnings, users continued through 70.2%of Google Chromes SSL warnings. This indicates thatthe user experience of a warning can have a significantimpact on user behavior. Based on our findings, we makerecommendations for warning designers and researchers.

1 Introduction

An oft-repeated maxim in the security community is thefutility of relying on end users to make security decisions.Felten and McGraw famously wrote, Given a choicebetween dancing pigs and security, the user will pickdancing pigs every time [21]. Herley elaborates [17],

Not only do users take no precautions againstelaborate attacks, they appear to neglect evenbasic ones. For example, a growing body ofmeasurement studies make clear that ...[users]are oblivious to security cues [26], ignore cer-tificate error warnings [30] and cannot tell legit-imate web-sites from phishing imitations [11].1

\ast The Mozilla Firefox experiments were implemented while the au-thor was an intern at Mozilla Corporation.

1Citations updated to match our bibliography.

The security communitys perception of the oblivioususer evolved from the results of a number of laboratorystudies on browser security indicators [5, 11, 13, 15, 26,30, 34]. However, these studies are not necessarily rep-resentative of the current state of browser warnings in2013. Most of the studies evaluated warnings that havesince been deprecated or significantly modified, often inresponse to criticisms in the aforementioned studies. Ourgoal is to investigate whether modern browser securitywarnings protect users in practice.

We performed a large-scale field study of user deci-sions after seeing browser security warnings. Our studyencompassed 25,405,944 warning impressions in GoogleChrome and Mozilla Firefox in May and June 2013. Wecollected the data using the browsers telemetry frame-works, which are a mechanism for browser vendors tocollect pseudonymous data from end users. Telemetryallowed us to unobtrusively measure user behavior duringnormal browsing activities. This design provides realism:our data reflects users actual behavior when presentedwith security warnings.

In this paper, we present the rates at which users clickthrough (i.e., bypass) malware, phishing, and SSL warn-ings. Low clickthrough rates are desirable because theyindicate that users notice and heed the warnings. Click-through rates for the two browsers malware and phish-ing warnings ranged from 9% to 23%, and users clickedthrough 33.0% of Mozilla Firefoxs SSL warnings. Thisdemonstrates that browser security warnings can effec-tively protect most users in practice.

Unfortunately, users clicked through Google ChromesSSL warning 70.2% of the time. This implies that theuser experience of a warning can have a significant impacton user behavior. We discuss several factors that mightcontribute to this warnings higher clickthrough rates. Ourpositive findings for the other five warnings suggest thatthe clickthrough rate for Google Chromes SSL warningcan be improved.

We also consider user behaviors that are indicative ofattention to warnings. We find that Google ChromesSSL clickthrough rates vary by the specific type of error.In Mozilla Firefox, a fifth of users who choose to clickthrough an SSL warning remove a default option, showingthey are making cognitive choices while bypassing thewarning. Together, these results contradict the stereotypeof the wholly oblivious user with no interest in security.

We conclude that users can demonstrate agency whenconfronted with browser security warnings. Users do notalways ignore security warnings in favor of their desiredcontent. Consequently, security experts and platformdesigners should not dismiss the role of the user. Wefind that the user experience of warnings can have anenormous impact on user behavior, justifying efforts tobuild usable warnings.

Contributions. We make the following contributions:

To our knowledge, we present the first in-depth,large-scale field study of browser security warnings.

We survey prior laboratory studies of browser secu-rity warnings and discuss why our field study datadiffers from prior research.

We analyze how demographics (operating systemand browser channel), warning frequency, and warn-ing complexity affect users decisions. Notably,we find evidence suggesting that technically skilledusers ignore warnings more often, and warning fre-quency is inversely correlated with user attention.

We provide suggestions for browser warning design-ers and make recommendations for future studies.

2 Background

Web browsers show warnings to users when an attackmight be occurring. If the browser is certain that an attackis occurring, it will show an error page that the user cannotbypass. If there is a chance that the perceived attack is afalse positive, the browser will show a bypassable warningthat discourages the user from continuing. We study onlybypassable warnings because we focus on user decisions.

A user clicks through a warning to dismiss it and pro-ceed with her original task. A user leaves the warningwhen she navigates away and does not continue with heroriginal task. A clickthrough rate describes the proportionof users who clicked through a warning type. When auser clicks through a warning, the user has (1) ignoredthe warning because she did not read or understand it or(2) made an informed decision to proceed because she be-lieves that the warning is a false positive or her computeris safe against these attacks (e.g., due to an antivirus).

Figure 1: Malware warning for Google Chrome

Figure 2: Malware warning for Mozilla Firefox

Figure 3: SSL warning for Google Chrome. The first paragraphchanges depending on the specific SSL error.

Figure 4: SSL warning for Mozilla Firefox

Figure 5: SSL Add Exception Dialog for Mozilla Firefox

We focus on three types of browser security warnings:malware, phishing, and SSL warnings. At present, allthree types of warnings are full-page, interstitial warningsthat discourage the user from proceeding.

2.1 Malware and Phishing Warnings

Malware and phishing warnings aim to prevent users fromvisiting websites that serve malicious executables or tryto trick users. Google Chrome and Mozilla Firefox relyon the Google Safe Browsing list [25] to identify mal-ware and phishing websites. The browsers warn usersaway from the sites instead of blocking them because theSafe Browsing service occasionally has false positives,although the false positive rate is very low [25].

Clickthrough Rate. If a malware or phishing warningis a true positive, clicking through exposes the user to adangerous situation. Nearly all Safe Browsing warningsare true positives; the false positive rate is low enoughto be negligible. The ideal clickthrough rate for malwareand phishing warnings is therefore close to 0%.

Warning Mechanisms. The browsers routinely fetch alist of suspicious (i.e., malware or phishing) sites fromSafe Browsing servers. If a user tries to visit a site thatis on the locally cached list, the browser checks with theSafe Browsing service that the URL is still on the malwareor phishing list. If the site is still on one of the lists, thebrowser presents a warning.

The two browsers behave differently if a page loadsa third-party resource (e.g., a script) from a URL on theSafe Browsing list. Google Chrome stops the page loadand replaces the page with a warning. Mozilla Firefoxblocks the third-party resource with no warning. As aresult, Mozilla Firefox users can see fewer warnings thanGoogle Chrome users, despite both browsers using thesame Safe Browsing list.

Warning Design. Figures 1 and 2 show the GoogleChrome and Mozilla Firefox warnings. Their phishingwarnings are similar to their respective malware warn-ings. When a browser presents the user with a malware orphishing warning, she has three options: leave the pagevia the warnings escape button, leave the page by closingthe window or typing a new URL, or click through thewarning and proceed to the page. The warnings also allowthe user to seek more information about the error.

Click Count. Mozilla Firefox users who want to bypassthe warning need to click one button: the Ignore thiswarning link at the bottom right. On the other hand,Chrome users who want to bypass the warning need toclick twice: first on the Advanced link, and then onProceed at your own risk.

2.2 SSL WarningsThe Secure Sockets Layer (SSL/TLS) protocol providessecure channels between browsers and web servers, mak-ing it fundamental to user security and privacy on theweb. As a critical step, the browser verifies a serversidentity by validating its public-key certificate against aset of trusted root authorities. This validation will fail inthe presence of a man-in-the-middle (MITM) attack.

Authentication failures can also occur in a wide vari-ety of benign scenarios, such as server misconfigurations.Browsers usually cannot distinguish these benign scenar-ios from real MITM attacks. Instead, browsers presentusers with a warning; users have the option to bypass thewarning, in case the warning is a false positive.

Clickthrough Rate. We hope for a 0% clickthrough ratefor SSL warnings shown during MITM attacks. However,many SSL warnings may be false positives (e.g., servermisconfigurations). There are two competing views re-garding SSL false positives. In the first, warning textshould discourage users from clicking through both trueand false positives, in order to incentivize developersto get valid SSL certificates. In the other, warning textshould provide users with enough information to correctlyidentify and dismiss false positives. The desired click-through rates for false-positive warnings would be 0%and 100%, respectively. In either case, false positives areundesirable for the user experience because we do notwant to annoy users with invalid warnings. Our goal istherefore a 0% clickthrough rate for all SSL warnings:users should heed all valid warnings, and the browsershould minimize the number of false positives.

Warning Design. Figures 3 and 4 present Google Chromeand Mozilla Firefoxs SSL warnings. The user can leavevia the warnings escape button, manually navigate away,or click through the warning. In Mozilla Firefox, theuser must also click through a second dialog (Figure 5) tobypass the warning.

The browsers differ in their presentation of the techni-cal details of the error. Google Chrome places informationabout the specific error in the main warning (Figure 3, firstparagraph), whereas Firefox puts the error informationin the hidden Technical Details section and the secondAdd Exception dialog (Figure 5).

Click Count. Mozilla Firefoxs SSL warning requiresmore clicks to bypass. Google Chrome users clickthrough a single warning button to proceed. On the otherhand, Mozilla Firefoxs warning requires three clicks:(1) click on I Understand The Risks, (2) click on theAdd Exception button, which raises a second dialog,(3) click on Confirm Security Exception in the seconddialog. By default, Firefox permanently remembers theexception and will not show the warning again if the userreencounters the same certificate for that website. Incontrast, Chrome presents the warning every time anddoes not remember the users past choices.

2.3 Browser Release ChannelsMozilla and Google both follow rapid release cycles.They release official versions of their browsers every sixor seven weeks, and both browsers update automatically.The official, default version of a browser is referred to asstable (Google Chrome) or release (Mozilla Firefox).

If users are interested in testing pre-release browserversions, they can switch to a different channel. The sta-ble/release channel is the recommended channel for endusers, but a minority of users choose to use earlier chan-nels to test cutting-edge features. The Beta channel isseveral weeks ahead of the stable/release channel. Thedeveloper (Google Chrome) or Aurora (Mozilla Fire-fox) channel is delivered even earlier. Both browsers alsooffer a nightly (Mozilla Firefox) or Canary (GoogleChrome) release channel, which updates every day andclosely follows the development repository.

The pre-release channels are intended for advancedusers who want to experience the latest-and-greatest fea-tures and improvements. They give website, extension,and add-on developers time to test their code on upcom-ing versions before they are deployed to end users. Theearly channels are not recommended for typical end usersbecause they can have stability issues, due to being un-der active development. The rest of this paper assumesa positive correlation between pre-release channels useand technical ability. While this matches the intentionof browser developers, we did not carry out any study tovalidate this assumption.

3 Prior Laboratory Studies

We survey prior laboratory studies of SSL and phishingwarnings. The body of literature paints a grim pictureof browser security warnings, but most of the warningshave since been deprecated or modified. In some cases,warnings were changed in response to these studies.

Only two studies evaluated warnings that are similarto the modern (June 2013) browser warnings that westudy in this paper. Sunshine et al. and Sotirakopouloset al. reported clickthrough rates of 55% to 80% for theFirefox 3 and 3.5 SSL warnings, which are similar butnot identical to the current Firefox SSL warning [29, 30].However, Sotirakopoulos et al. concluded that laboratorybiases had inflated both studies clickthrough rates [29].

3.1 SSL Warnings

SSL warnings are the most studied type of browser warn-ing. Usability researchers have evaluated SSL warningsin both SSL-specific studies and phishing studies becauseSSL warnings and passive indicators were once viewedas a way to identify phishing attacks.2

Dhamija et al. performed the first laboratory study ofSSL warnings in 2006. They challenged 22 study partic-ipants to differentiate between phishing and legitimatewebsites in Mozilla Firefox 1.0.1 [11]. In this version,the warning was a modal dialog that allowed the user topermanently accept, temporarily accept, or reject the cer-tificate. When viewing the last test website, participantsencountered an SSL warning. The researchers reportedthat 15 of their 22 subjects (68%) quickly clicked throughthe warning without reading it. Only one user was laterable to tell the researchers what the warning had said.The authors considered the clickthrough rate of 68% aconservative lower bound because participants knew thatthey should be looking for security indicators.

In 2007, Schechter et al. studied user reactions to Inter-net Explorer 7s SSL warning, which is the same one-clickinterstitial that is present in all subsequent versions of In-ternet Explorer [26]. Participants encountered the warningwhile logging into a bank website to look up information.The researchers were aware of ecological validity con-cerns with laboratory studies and split their participantsinto three groups: participants who entered their own cre-dentials, a role-playing group that entered fake passwords,and a security-primed role-playing group that enteredfake passwords. Overall, 53% of the total 57 participantsclicked through. However, only 36% of the non-role-playing group clicked through. The difference betweenthe role-playing participants and non-role-playing partic-

2There is evidence that modern phishing sites can have valid SSLcertificates [24].

ipants was statistically significant, illustrating one chal-lenge of experiments in artificial environments.

Sunshine et al. performed multiple studies of SSL warn-ings in 2009 [30]. First, they conducted an online survey.They asked 409 people about Firefox 2, Firefox 3, andInternet Explorer 7 warnings. Firefox 2 had a modal dia-log like Firefox 1.0.1, and Firefox 3s warning is similarbut not identical to the current Firefox warning. Lessthan half of respondents said they would continue to thewebsite after seeing the warning. As a follow-up, Sun-shine et al. also conducted a laboratory study that exposed100 participants to SSL warnings while completing infor-mation lookup tasks. The clickthrough rates were 90%,55%, and 90% when participants tried to access their bankwebsites in Firefox 2, Firefox 3, and Internet Explorer7, respectively. The clickthrough rates increased to 95%,60%, and 100% when participants saw an SSL warningwhile trying to visit the university library website.

Sotirakopoulos et al. replicated Sunshines laboratorySSL study with a more representative population sam-ple [29]. In their study, 80% of participants using Firefox3.5 and 72% of participants using Internet Explorer 7clicked through an SSL warning on their bank website.More than 40% of their participants said that the labora-tory environment had influenced them to click throughthe warnings, either because they felt safe in the studyenvironment or were trying to complete the experimentaltask. Sotirakopoulos et al. concluded that the laboratoryenvironment biased their results, and they suspect thatthese biases are also present in similar laboratory studies.

Bravo-Lillo et al. interviewed people about an SSLwarning from an unspecified browser [5]. They asked 20participants about the purpose of the warning, what wouldhappen if a friend were to click through, and whether afriend should click through the warning. Participantswere separated into advanced and novice browserusers. Advanced participants said they would not clickthrough an SSL warning on a bank website, but noviceparticipants said they would.

Passive Indicators. Some studies focused on passiveSSL indicators, which non-interruptively show the statusof the HTTP(S) connection in the browser UI. Althoughbrowsers still have passive SSL indicators, interruptiveSSL and phishing warnings are now the primary tool forcommunicating security information to users.

Friedman et al. asked participants whether screenshotsof websites depicted secure connections; many partici-pants could not reliably determine whether a connectionwas secure [15]. Whalen and Inkpen used eye-trackingsoftware to determine that none of their 16 participantslooked at the lock or key icon in the URL bar, HTTP(S)status in the URL bar, or the SSL certificate when asked tobrowse websites normally [33]. Some browsers modify

the lock icon or color of the URL bar to tell the user whena website has an Extended Validation (EV) SSL certifi-cate. Jackson et al. asked 27 study subjects to classify12 websites as either phishing or legitimate sites, but theEV certificates did not help subjects identify the phish-ing sites [19]. In a follow-up study, Sobey et al. foundthat none of their 28 subjects clicked on the EV indica-tors, and the presence of EV indicators did not affectdecision-making [28]. Similarly, Biddle et al. found thatstudy participants did not understand Internet Explorerscertificate summaries [3].

In 2012, a Google Chrome engineer mentioned highclickthrough rates for SSL warnings on his blog [20]. Weexpand on this with a more accurate and detailed view ofSSL clickthrough rates in Google Chrome.

3.2 Phishing Warnings

Phishing warnings in contemporary browsers are active,interstitial warnings; in the past, they have been passiveindicators in toolbars. Researchers have studied whetherthey are effective at preventing people from entering theircredentials into phishing websites.

Wu et al. studied both interstitial and passive phish-ing warnings [34]. Neither of the warnings that theyevaluated are currently in use in browsers. First, theylaunched phishing attacks on 30 participants. The par-ticipants role-played during the experiment while usingsecurity toolbars that display passive phishing warnings.Despite the toolbars, at least one attack fooled 20 outof 30 participants. In their next experiment, they asked10 study participants to perform tasks on PayPal and ashopping wish list website; they injected modal phishingwarnings into the websites. None of the subjects enteredthe credentials into the PayPal site, but the attack on thewish list site fooled 4 subjects. The authors do not reportthe warning clickthrough rates.

Egelman et al. subjected 60 people to simulated phish-ing attacks in Internet Explorer 7 or Mozilla Firefox2.0 [13]. Firefox 2.0 had a modal phishing dialog that isnot comparable to the current Mozilla Firefox phishingdialog, and Internet Explorer had both passive and activewarnings. Participants believed that they were taking partin a laboratory study about shopping. The researchersasked participants to check their e-mail, which containedboth legitimate shopping confirmation e-mails and similarspear phishing e-mails sent by the researchers. Users whoclicked on the links in the phishing e-mails saw a phishingwarning. Participants who saw Mozilla Firefoxs activewarning, Internet Explorers active warning, or InternetExplorers passive warning were phished 0%, 45%, and90% of the time, respectively. The clickthrough rateswere an unspecified superset of the rates at which peoplefell for the phishing attacks.

3.3 Malware Download WarningsGoogle Chrome and Microsoft Internet Explorer also dis-play non-blocking warning dialogs when users attemptto download malicious executables. In a blog post, a Mi-crosoft employee stated that the clickthrough rate for Inter-net Explorers SmartScreen warning was under 5% [16].We did not study this warning for Google Chrome, andMozilla Firefox does not have this warning.

4 Methodology

We rely on the telemetry features implemented in MozillaFirefox and Google Chrome to measure clickthrough ratesin situ. Telemetry is a mechanism for browser vendors tocollect pseudonymous data from end users who opt in tostatistics reporting. Google Chrome and Mozilla Firefoxuse similar telemetry platforms.

4.1 Measuring Clickthrough RatesWe implemented metrics in both browsers to count thenumber of times that a user sees, clicks through, or leavesa malware, phishing, or SSL warning. Based on this data,we can calculate clickthrough rates for each warning type.As discussed in Section 2, we report only the clickthroughrates for warnings that the user can bypass. We measuredthe prevalence of non-bypassable warnings separately. Tosupplement the clickthrough rates, we recorded whetherusers clicked on links like Help me understand, View,or Technical Details.

Bypassing some warnings takes multiple clicks, andour clickthrough rates for these warnings represent thenumber of users who completed all of the steps to proceedto the page. For Mozilla Firefoxs SSL warning (whichtakes three clicks to proceed), we recorded how oftenusers perform two intermediate clicks (on Add Excep-tion or Confirm Security Exception) as well as theoverall clickthrough rate.

We also measured how often users encounter and clickthrough specific SSL errors. In addition to the overallclickthrough rates for the warnings, we collected click-through data for each type of Mozilla Firefox SSL errorand the three most common Google Chrome SSL errors.

Our Mozilla Firefox data set does not allow us to trackspecific telemetry participants. In Google Chrome, wecan correlate warning impressions with psuedonymousbrowser client IDs; however, the sample size for mostindividual users is too small to draw conclusions. Wetherefore report the results of measurements aggregatedacross all users unless otherwise specified. The telemetryframeworks do not provide us with any personal or demo-graphic information except for the operating system andbrowser version for each warning impression.

4.2 Measuring Time Spent on WarningsWe also used the Google Chrome telemetry framework toobserve how much time Google Chrome users spent onSSL warnings. Timing began as soon as an SSL warningcame to the foreground in a tab. In particular,

We recorded the time spent on a warning and associ-ated it with the outcome (click through or leave).

We recorded the time spent on a warning and associ-ated it with the error type, if it was one of the threemost common error types (untrusted authority, namemismatch, and expired certificate).

Together, these correspond to five timing measurements(two for outcome and three for error type). For scalability,the telemetry mechanism in Google Chrome only allowstiming measurements in discrete buckets. As a result, ouranalysis also treats time as a discrete, ordinal variable.

We used log-scaled bucket sizes (e.g., the first bucketsize is 45ms but the last is 90,279ms) with 50 buckets,ranging from 0ms to 1,200,000ms, for the two outcomehistograms. The three error type histograms had 75 buck-ets each, ranging from 0ms to 900,000ms. We used morebuckets for the error histograms because we anticipatedthat they would be more similar to each other.

4.3 EthicsWe collected data from users who participate in theirbrowsers broad, unpaid user metrics programs. At firstrun of a browser, the browser asks the user to share usagedata. If the user consents, the browser collects data onperformance, features, and stability. In some pre-releasedeveloper channels, data collection is enabled by default.The browser periodically sends this pseudonymous dataover SSL to the central Mozilla or Google servers foranalysis. The servers see the IP addresses of clients bynecessity, but they are not attached to telemetry data. Alltelemetry data is subject to strict privacy policies andparticipants can opt out by changing their settings [7, 23].Multiple Google Chrome committers and Mozilla Firefoxcontributors reviewed the data collection code to ensurethat the metrics did not collect any private data.

This work is not considered human subjects research byUC Berkeley because the student did not have access todatabase identifiers or personally identifying information.

4.4 Data Collection

Collection Period. Google Chromes malware and phish-ing measurement code was in place in Chrome 24 priorto our work, and our SSL measurement code was addedto Google Chrome 25. The Google Chrome data in this

paper was collected April 28 - May 31, 2013. Our MozillaFirefox measurement code was added to Firefox 17, anda bug in the SSL measurement code was fixed in Firefox23. The data on the Firefox malware warning, phishingwarning, and SSL Add Exception dialog was collectedMay 1-31, 2013. The data on Firefox SSL warnings wascollected June 1 - July 5, 2013, as the Firefox 23 fixprogressed through the various release channels.

Sample Sizes. In Google Chrome, we recorded 6,040,082malware warning impressions, 386,350 phishing warningimpressions, and 16,704,666 SSL warning impressions.In Mozilla Firefox, we recorded 2,163,866 malware warn-ing impressions, 100,004 phishing warning impressions,and 10,976 SSL warning impressions. Appendix A fur-ther breaks downs these sample sizes by OS and channel.

Number of Users. For Mozilla Firefox, we recordedwarning impressions from the approximately 1% of Fire-fox users who opt in to share data with Mozilla via teleme-try. In Google Chrome, we observed malware, phishing,and SSL warning impressions on 2,148,026; 204,462; and4,491,767 clients (i.e., browser installs), respectively.

4.5 Method Limitations

Private Data. Due to privacy constraints, we could notcollect information about users personal demographicsor browsing habits. Consequently, we cannot measurewhether user behavior differs based on personal character-istics, the target site, or the source of the link to the site.We also cannot identify SSL false positives due to captiveportals, network proxies, or server misconfigurations.

Sampling Bias. The participants in our field study are nota random population sample. Our study only representsusers who opt in to browser telemetry programs. Thismight present a bias. The users who volunteered might bemore likely to click through dialogs and less concernedabout privacy. Thus, the clickthrough rates we measurecould be higher than population-wide rates. Given thatmost of our observed rates are low, this bias augments ourclaim that clickthrough rates are lower than anticipated.

Overrepresentation. We present clickthrough ratesacross all warnings shown to all users. A subset of userscould potentially be overrepresented in our analysis.Within the Google Chrome data set, we identified andremoved a small number of overrepresented clients whowe believe are either crawlers or malware researchers.We were unable to remove individual clients from theMozilla Firefox set, but we do not believe this representsa bias because we know that the overrepresented clientsin Chrome still contributed fewer than 1% of warningimpressions. Some clients experienced multiple typesof warning impressions; we investigated this in Chrome

and found that the clickthrough rates do not differ ifwe remove non-independent clients. Our large samplesizes and small critical value (\alpha = 0.001) should furtherameliorate these concerns.

Frames. Our original measurement for Mozilla Firefoxdid not differentiate between warnings shown in top-levelframes (i.e., warnings that fill the whole tab) and warningsshown in iframes. In contrast, Google Chrome alwaysshows malware and phishing warnings in the top-levelframe and does not render any warning type in iframes.Since users might not notice warnings in iframes, the twometrics are not necessarily directly comparable.

Upon discovering this issue, we modified our Firefoxmeasurement implementation to take frame level intoaccount. Our new implementation is not available to allFirefox users yet, but we have data for recent pre-releasechannels. For malware and phishing warning impressionscollected from the beta channel, the clickthrough rate forthe top-level frame is within two percentage points ofthe overall clickthrough rate. This is due to the relativeinfrequency of malware and phishing warnings in iframesand the low overall clickthrough rate. Since the framelevel does not make a notable difference for malware andphishing warnings, we present the overall rates (includingboth top-level frames and iframes) for the full samplesizes in Section 5.1. The difference is more importantfor SSL warnings: the clickthrough rate for top-levelframes is 28.7 percentage points higher than the overallclickthrough rate of 4.3%. Consequently, Section 5.2presents only the top-level frame rate for SSL warnings,although it limits our sample to pre-release users.

5 Clickthrough Rates

We present the clickthrough data from our measurementstudy. Section 5.1 discusses malware and phishing warn-ings together because they share a visual appearance. Wethen present rates for SSL warnings in Section 5.2.

5.1 Malware and Phishing Warnings

The clickthrough rates for malware warnings were 7.2%and 23.2% in stable versions of Mozilla Firefox andGoogle Chrome, respectively. For phishing warnings,we found clickthrough rates of 9.1% and 18.0%. In thissection, we discuss the effects of warning type, demo-graphics, and browser on the clickthrough rates.

5.1.1 Malware Rates by Date

The malware warning clickthrough rates for GoogleChrome vary widely by date. We have observed click-through rates ranging from 11.2% to 24.9%, depending

Operating Malware PhishingSystem Firefox Chrome Firefox Chrome

Windows 7.1% 23.5% 8.9% 17.9%MacOS 11.2% 16.6% 12.5% 17.0%Linux 18.2% 13.9% 34.8% 31.0%

Table 1: User operating system vs. clickthrough rates for mal-ware and phishing warnings. The data comes from stable (i.e.,release) versions.

Channel Malware PhishingFirefox Chrome Firefox ChromeStable 7.2% 23.2% 9.1% 18.0%Beta 8.7% 22.0% 11.2% 28.1%Dev 9.4% 28.1% 11.6% 22.0%Nightly 7.1% 54.8% 25.9% 20.4%

Table 2: Release channel vs. clickthrough rates for malware andphishing warnings, for all operating systems.

on the week, since the current version of the warningwas released in August 2012. In contrast, the MozillaFirefox malware warning clickthrough rate across weeksstays within one percentage point of the month-longaverage. We did not observe similar temporal variationsfor phishing or SSL warnings.

Recall from Section 2.1 that Google Chrome andMozilla Firefoxs malware warnings differ with respectto secondary resources: Google Chrome shows aninterstitial malware warning if a website includessecondary resources from a domain on the Safe Browsinglist, whereas Mozilla Firefox silently blocks the resource.We believe that this makes Google Chromes malwareclickthrough rates more sensitive to the contents of theSafe Browsing list. For example, consider the case wherea well-known website accidentally loads an advertisementfrom a malicious domain. Google Chrome would showa warning, which users might not believe because theytrust the website. Mozilla Firefox users would not seeany warning. Furthermore, Chrome phishing warningsare less likely to be due to secondary resources, and thatwarnings clickthrough rates do not vary much by time.

5.1.2 Malware/Phishing Rates by Warning Type

In Mozilla Firefox, we find a significantly higher click-through rate for phishing warnings than malware warn-ings (\chi 2 test: p(1) < 0.0001). This behavior is rational:a malware website can infect the users computer withoutany action on the users part, but a phishing website canonly cause harm by tricking the user at a later point intime. Mozilla Firefox makes this priority ordering explicitby choosing to display the malware warning if a website

is listed as both malware and phishing.3 However, thepractical difference is small: 7.2% vs. 9.1%.

In Google Chrome, the average malware clickthroughrate is higher than the phishing clickthrough rate. How-ever, the malware clickthrough rate fluctuates widely (Sec-tion 5.1.1); the malware clickthrough rate is sometimeslower than the phishing clickthrough rate.

5.1.3 Malware/Phishing Rates by Demographics

We consider whether users of different operating systemsand browser release channels react differently to warn-ings. As Table 1 depicts, Linux users have significantlyhigher clickthrough rates than Mac and Windows userscombined for the Firefox malware warning, Firefox phish-ing warning, and Chrome phishing warning (\chi 2 tests:p(1) < 0.0001). While the low prevalence of malwarefor Linux could explain the higher clickthrough rates forthe Firefox malware warning, use of Linux does not pro-vide any additional protection against phishing attacks.The Chrome malware warning does not follow the samepattern: Windows users have a significantly higher click-through rate (\chi 2 tests: p(1) < 0.0001).

We also see differences between software releasechannels (Table 2). Nightly users click through GoogleChrome malware and Firefox phishing warnings at muchhigher rates than stable users, although they click throughFirefox malware and Google Chrome phishing warningsat approximately the same rates.

In several cases, Linux users and early adopters clickthrough malware and phishing warnings at higher rates.One possible explanation is that a greater degree of tech-nical skill as indicated by use of Linux or early-adopterversions of browsers corresponds to reduced risk aver-sion and an increased willingness to click through warn-ings. This does not hold true for all categories and warn-ings (e.g., nightly and stable users click through the Fire-fox malware warning at the same rate), suggesting theneed for further study.

5.1.4 Malware/Phishing Rates by Browser

Google Chrome stable users click through phishing warn-ings more often than Mozilla Firefox stable users. Thisholds true even when we account for differences in howthe browsers treat iframes (Section 4.5). Mozilla Fire-foxs beta channel users still click through warnings at alower rate when we exclude iframes: 9.6% for malwarewarnings, and 10.8% for phishing warnings.

One possibility is that Mozilla Firefoxs warnings aremore frightening or more convincing. Another possi-

3Google Chrome will display both warnings. To preserve inde-pendence, our measurement does not include any warnings with bothphishing and malware error messages. Dual messages are infrequent.

bility is that the browsers have different demographicswith different levels of risk tolerance, which is reflectedin their clickthrough rates. There might be differencesin technical education, gender, socioeconomic status, orother factors that we cannot account for in this study. Insupport of this theory, we find that differences betweenthe browsers do not hold steady across operating systemsor channels. The gap between the browsers narrows orreverses for some categories of users, such as Linux usersand nightly release users.

5.2 SSL WarningsThe clickthrough rates for SSL warnings were 33.0% and70.2% for Mozilla Firefox (beta channel) and GoogleChrome (stable channel), respectively.

5.2.1 SSL Rates by Demographic

In Section 5.1, we observed that malware and phishingclickthrough rates differed across operating systems andchannels. For SSL, the differences are less pronounced.

As with the malware and phishing warnings, nightlyusers click through SSL warnings at a higher rate for bothFirefox and Chrome (\chi 2 tests: p < 0.0001).

The effect of users operating systems on SSL click-through rates differs for the two browsers. In Firefox,Linux users are much more likely to click through SSLwarnings than Windows and Mac users combined (\chi 2 test:p < 0.0001), although it is worth noting that the FirefoxLinux sample size is quite small (58). In Chrome, Win-dows users are very slightly more likely to click throughSSL warnings than Linux and Mac users combined (\chi 2

test: p < 0.0001).

5.2.2 SSL Rates by Browser

We find a large difference between the Mozilla Firefoxand Google Chrome clickthrough rates: Google Chromeusers are 2.1 times more likely to click through an SSLwarning than Mozilla Firefox users. We explore fivepossible causes.

Number of Clicks. Google Chrome users click one but-ton to dismiss an SSL warning, but Mozilla Firefox usersneed to click three buttons. It is possible that the addi-tional clicks deter people from clicking through. However,we do not believe this is the cause of the rate gap.

First, the number of clicks does not appear to affectthe clickthrough rates for malware and phishing warn-ings. Mozilla Firefoxs malware and phishing warningsrequire one click to proceed, whereas Google Chromesmalware and phishing warnings require two. The GoogleChrome malware and phishing warnings with two clicksdo not have lower clickthrough rates than the MozillaFirefox warnings with one click. Second, as we discussin Section 5.2.3, 84% of users who perform the first two

Operating SSL WarningsSystem Firefox Chrome

Windows 32.5% 71.1%MacOS 39.3% 68.8%Linux 58.7% 64.2%Android NC 64.6%

Table 3: User operating system vs. clickthrough rates for SSLwarnings. The Google Chrome data is from the stable channel,and the Mozilla Firefox data is from the beta channel.

Channel SSL WarningsFirefox ChromeRelease NC 70.2%Beta 32.2% 73.3%Dev 35.0% 75.9%Nightly 43.0% 74.0%

Table 4: Channel vs. clickthrough rates for SSL warnings.

clicks in Mozilla Firefox also perform the third. Thisindicates that the extra click is not a determining deci-sion point. Unfortunately, we do not have data on thedifference between the first and second clicks.

Warning Appearance. The two warnings differ in sev-eral ways. Mozilla Firefoxs warning includes an imageof a policeman and uses the word untrusted in the title.These differences likely contribute to the rate gap. How-ever, we do not think warning appearance is the sole orprimary factor; the browsers malware and phishing warn-ings also differ, but there is only about a 10% differencebetween browsers for these warnings.

Certificate Pinning. Google Chrome ships with a listof pinned certificates and preloaded HTTP Strict Trans-port Security (HSTS) sites. Users cannot click throughSSL warnings on sites protected by these features. Certifi-cate pinning and HSTS cover some websites with impor-tant private data such as Google, PayPal, and Twitter [8].In contrast, Mozilla Firefox does not come with manypreloaded pinned certificates or any pre-specified HSTSsites. As a result, Chrome shows more non-bypassablewarnings: our field study found that 20% of all GoogleChrome SSL warning impressions are non-bypassable, ascompared to 1% for Mozilla Firefox.

Based on this, we know that Mozilla Firefox users seemore warnings for several critical websites. If we assumethat users are less likely to click through SSL warningson these critical websites, then it follows that MozillaFirefoxs clickthrough rate will be lower. This potentialbias could account for up to 15 points of the 37-point gapbetween the two clickthrough rates, if we were to assumethat Google Chrome users would never click through SSLerrors on critical websites if given the chance.

Remembering Exceptions. Due to the permanentlystore this exception feature in Mozilla Firefox, MozillaFirefox users see SSL warnings only for websites withoutsaved exceptions. This means that Mozilla Firefox usersmight ultimately interact with websites with SSL errorsat the same rate as Google Chrome users despite havinglower clickthrough rates. For example, imagine a userthat encounters two websites with erroneous SSL config-uration: she leaves the first after seeing a warning, butvisits the second website nine times despite the warning.This user would have a 50% clickthrough rate in MozillaFirefox but a 90% clickthrough rate in Google Chrome,despite visiting the second website at the same rate.

We did not measure how often people revisit websiteswith SSL errors. However, we suspect that people dorepeatedly visit sites with warnings (e.g., a favorite sitewith a self-signed certificate). If future work were toconfirm this, there could be two implications. First, ifusers are repeatedly visiting the same websites with errors,the errors are likely false positives; this would mean thatthe lack of an exception-storing mechanism noticablyraises the false positive rate in Google Chrome. Second,warning fatigue could be a factor. If Google Chrome usersare exposed to more SSL warnings because they cannotsave exceptions, they might pay less attention to eachwarning that they encounter.

Demographics. Its possible that the browsers have dif-ferent demographics with different levels of risk toler-ance. However, this factor likely only accounts for afew percentage points because the same demographic ef-fect applies to malware and phishing warnings, and thedifference between browsers for malware and phishingwarnings is much smaller.

5.2.3 SSL Rates by Certificate Error Type

To gain insight into the factors that drive clickthroughrates, we study whether the particular certificate erroraffects user behavior.

Google Chrome. Google Chromes SSL warning in-cludes a short explanation of the particular error, andclicking on Help me understand will open a more-detailed explanation. In case a certificate has multipleerrors, Google Chrome only shows the first error out ofuntrusted issuer error, name mismatch error, and certifi-cate expiration error, respectively.

Table 5 presents the clickthrough rates by error typesfor Google Chrome. If Google Chrome users are payingattention to and understanding the warnings, one wouldexpect different clickthrough rates based on the warningtypes. We find a 24.4-point difference between the click-through rates for untrusted issuer errors and expired certifi-cate errors. One explanation could be that untrusted issuer

Certificate Error Percentageof Total

ClickthroughRate

Untrusted Issuer 56.0% 81.8%Name Mismatch 25.0% 62.8%Expired 17.6% 57.4%Other Error 1.4% All Error Types 100.0% 70.2%

Table 5: Prevalence and clickthrough rates of error types for theGoogle Chrome SSL warning. Google Chrome only displaysthe most critical warning; we list the error types in order, withuntrusted issuer errors as the most critical. Data is for the stablechannel across all operating systems.

errors appear on unimportant sites, leading to higher click-through rates without user attention or comprehension;however, the Mozilla Firefox data suggests otherwise.An alternative explanation could be that expired certifi-cates, which often occur for websites with previouslyvalid certificates [1], surprise the user. In contrast, un-trusted certificate errors always occur for a website andconform with expectations.

Mozilla Firefox. Mozilla Firefoxs SSL warning does notinform the user about the particular SSL error by default.4

Instead, the secondary Add Exception dialog presentsall errors in the SSL certificate. The user must confirmthis dialog to proceed.

Table 6 presents the rates at which users confirm theAdd Exception dialog in Mozilla Firefox. The errortypes do not greatly influence the exception confirmationrate. This indicates that the Add Exception dialog doesnot do an adequate job of explaining particular error cate-gories and their meaning to the users. Thus, users ignorethe categories and click through errors at the same rate.This finding also suggests that the differences in click-through rates across error types in Google Chrome cannotbe attributed to untrusted issuer errors corresponding tounimportant websites; if that were the case, we wouldexpect to see the same phenomenon in Firefox.

Error Prevalence. The frequency of error typesencountered by users in our field study also indicates thebase rate of SSL errors on the web. Our Google Chromedata contradicts a previous network telemetry study,which suggested that untrusted issuer errors correspondto 80% of certificate errors seen on the wire [18]. Also,Google Chrome users see fewer untrusted issuer errorsthan Mozilla Firefox users; this may be because MozillaFirefox users are more likely to click on the AddException dialog for untrusted issuer errors. Recall thatwe collect the Mozilla Firefox error type statistics onlyafter a user clicks on the Add Exception button.

4This information is available under the Technical details link, butour measurements indicate that it is rarely opened (Section 5.2.4).

Certificate Error Percentageof Total

ConfirmationRate

Untrusted Issuer 38% 87.1%Untrusted and NameMismatch

26.4% 87.9%

Name Mismatch 15.7% 80.3%Expired 10.2% 80.7%Expired, Untrusted andName Mismatch

4.7% 87.6%

Expired and Untrusted 4.1% 83.6%Expired and Name Mis-match

0.7% 85.2%

None of the above

Date Invalid Name Invalid Authority Invalid

Untrusted Issuer

Name Mismatch

Expired

0400445495550611679755839933103711531282142515841761195721752418268829883321369141034560506856336261695977358597955610621118051312114584162101801720026222592474127499305653397337761419714665051851576326405871200791388796197768108669120785134252149220165857184349204903227748253140281363312733347601386356429432477311530528589678655423728498809721900000

29726 57841 181330 0.0195129206 0.0172819263 0.01866638871162 2575 8338 0.0008972521 0.0007693671 0.00072967581488 3179 10348 0.0011135482 0.0009498322 0.00093438692048 4330 13327 0.0014341184 0.0012937318 0.00128603792586 6129 17672 0.0019016838 0.001831243 0.00162387413679 9237 26943 0.0028993361 0.0027598616 0.00231022155722 16407 49754 0.0053540277 0.004902138 0.00359311979773 31289 98207 0.0105680549 0.0093486314 0.006136937917250 56722 189550 0.0203974747 0.0169475877 0.010832106728694 91402 318968 0.0343241452 0.0273093934 0.018018346141621 131258 468412 0.0504058134 0.0392177016 0.026135832756998 167760 594135 0.063934865 0.0501238905 0.035791792569097 190512 671696 0.0722812039 0.0569218087 0.043389337978215 199358 691696 0.0744334038 0.059564846 0.049114969883027 196592 667097 0.0717863055 0.0587384113 0.052136656682860 186615 604480 0.0650480904 0.0557574501 0.052031789380363 175240 536431 0.0577253378 0.0523587898 0.050463802676433 161351 471226 0.050708628 0.0482089882 0.047995966172087 146299 412341 0.0443720134 0.0437117016 0.045266903167063 132085 356626 0.0383765224 0.0394647954 0.042112091361199 118225 310620 0.033425817 0.0353236585 0.038429802955696 106502 271295 0.0291940539 0.0318210216 0.034974203950183 97773 237722 0.0255812635 0.029212942 0.031512325445687 87280 207645 0.0223446776 0.0260778086 0.028689070240904 81185 182721 0.0196626061 0.0242567242 0.025685593936318 75563 160877 0.0173119733 0.0225769643 0.022805823333073 70029 141367 0.0152125023 0.0209234974 0.020768131429324 64795 124313 0.0133773214 0.0193596655 0.018413953526578 60270 110218 0.0118605585 0.018007671 0.016689607724143 55668 97779 0.0105219978 0.0166326701 0.015160553821750 52553 87571 0.009423515 0.01570196 0.013657873719838 48189 78589 0.008456962 0.0143980696 0.012457236717971 44742 70654 0.0076030767 0.0133681635 0.011284857416648 40896 64167 0.0069050106 0.0122190428 0.010454081915652 37736 59229 0.0063736325 0.0112748875 0.009828645514865 34761 55095 0.0059287727 0.0103860071 0.009334450214425 32274 52055 0.0056016383 0.009642933 0.00905815313620 29410 50314 0.0054142893 0.0087872176 0.008552654713533 26744 47125 0.0050711211 0.0079906612 0.008498023213308 23974 41911 0.0045100425 0.0071630314 0.008356734913388 21045 38018 0.0040911168 0.0062878951 0.008406970713141 18499 34747 0.0037391245 0.0055271927 0.008251867513373 16232 32779 0.0035273481 0.0048498509 0.008397551513463 14308 30413 0.0032727428 0.0042749918 0.008454066813537 12399 28010 0.003014156 0.0037046144 0.00850053512989 11176 25100 0.0027010109 0.0033392024 0.008156419413014 9665 22125 0.0023808712 0.0028877408 0.008172118112640 8486 20483 0.0022041755 0.0025354753 0.007937265511783 7531 19297 0.0020765501 0.0022501372 0.007399113810720 6725 17605 0.001894474 0.0020093179 0.00673160499677 5985 15860 0.0017066945 0.0017882182 0.00607665498442 5512 14535 0.0015641113 0.0016468937 0.00530113887709 4859 13183 0.0014186226 0.0014517882 0.00484085287036 4433 12065 0.0012983146 0.0013245065 0.00441824376123 4100 10953 0.0011786523 0.0012250116 0.00384492695295 3751 10454 0.0011249549 0.0011207362 0.00332498584576 3417 9439 0.0010157308 0.0010209426 0.0028734914073 3169 8936 0.0009616029 0.0009468444 0.00255763313315 2866 8059 0.000867229 0.000856313 0.00208164832981 2658 7711 0.0008297807 0.0007941661 0.00187191362619 2626 7004 0.0007537004 0.000784605 0.00164459642275 2370 6777 0.0007292729 0.0007081165 0.00142858222101 2174 6307 0.0006786962 0.0006495549 0.00131931921817 2129 6049 0.0006509329 0.0006361097 0.00114098191567 2080 5759 0.000619726 0.0006214693 0.00098399491409 1840 5464 0.000587981 0.0005497613 0.0008847791258 1765 5189 0.0005583883 0.0005273526 0.00078995891131 1654 4816 0.0005182497 0.0004941876 0.00071020941085 1582 4793 0.0005157747 0.0004726752 0.00068132381089 1558 4535 0.0004880113 0.0004655044 0.0006838356885 1532 4442 0.0004780036 0.0004577361 0.0005557342851 1416 4200 0.000451962 0.0004230772 0.0005343839766 1322 4037 0.0004344216 0.0003949916 0.0004810083741 1293 3829 0.0004120387 0.0003863268 0.0004653096

11012 24265 81127 0.0087300761 0.0072499774 0.0069149658

0%1%2%3%4%5%6%7%8%

0

550

839

1282

1957

2988

4560

6959

1062

1

1621

0

2474

1

3776

1

Chart 2Untrusted IssuerName MismatchExpired

Figure 7: Google Chrome SSL clickthrough times (ms), by errortype. The graph shows the percent of warning impressions thatfall in each timing bucket. The x-axis increases logarithmically,and we cut off the distribution at 90% due to the long tail.

users who click through and proceed to the page. 47% ofusers who clicked through the warning made the decisionwithin 1.5s, whereas 47% of users who left the page did sowithin 3.5s. We interpret this to mean that users who clickthrough the warning often do so after less consideration.

6.2 Time by Error TypeFigure 7 depicts the click times for three error types (un-trusted authority, name mismatch, and expired certificateerrors). Users clicked through 49% of untrusted issuerwarning impressions within 1.7s, but clicked through 50%of name and date errors within 2.2s and 2.7s, respectively.We believe that this data is indicative of warning fatigue:users click through more-frequent errors more quickly.The frequency and clickthrough rate of each error type(as reported in Section 5.2) are inversely correlated withthat error types timing variance and mode (Figure 7).

7 Implications

Our primary finding is that browser security warnings canbe effective security mechanisms in practice, but theireffectiveness varies widely. This should motivate moreattention to improving security warnings. In this section,we summarize our findings and their implications, presentsuggestions for warning designers, and make recommen-dations for future warning studies.

7.1 Warning Effectiveness7.1.1 Clickthrough Rates

Popular opinion holds that browser security warnings areineffective. However, our study demonstrates that browser

security warnings can be highly effective at preventingusers from visiting websites: as few as a tenth of usersclick through Firefoxs malware and phishing warnings.We consider these warnings very successful.

We found clickthrough rates of 18.0% and 23.2% forGoogle Chromes phishing and malware warnings, re-spectively, and 31.6% for Firefoxs SSL warning. Thesewarnings prevent 70% (or more) of attempted visits topotentially dangerous websites. Although these warningscould be improved, we likewise consider these warningssuccessful at persuading and protecting users.

Google Chromes SSL warning had a clickthrough rateof 70.2%. Such a high clickthrough rate is undesirable:either users are not heeding valid warnings, or the browseris annoying users with invalid warnings and possibly caus-ing warning fatigue. Our positive findings for the otherwarnings demonstrate that this warning has the poten-tial for improvement. We hope that this study motivatesfurther studies to determine and address the cause of itshigher clickthrough rate. We plan to test an exception-remembering feature to investigate the influence of repeatexposures to warnings. At Google, we have also begun aseries of A/B tests in the field to measure the impact of anumber of improvements.

7.1.2 User Attention

Although we did not directly study user attention, tworesults of our study suggest that at least a minority ofusers pay attention to browser security warnings.

There is a 24.4-point difference between the click-through rates for untrusted issuer errors (81.8%) andexpired certificate errors (57.4%) in Google Chrome.

21.3% of the time that Mozilla Firefox users viewedthe Add Exception dialog, they un-checked thedefault Permanently store this exception option.

These results contradict the stereotype of wholly obliv-ious users with no interest in security.

7.2 Comparison with Prior ResearchAs Bravo-Lillo et al. wrote [5]:

Evidence from experimental studies indicatesthat most people dont read computer warnings,dont understand them, or simply dont heedthem, even when the situation is clearly haz-ardous.

In contrast, a majority of users heeded five of the sixtypes of browser warnings that we studied. This sectionexplores why our results differ from prior research.

Browser Changes. Most prior browser research was con-ducted between 2002 and 2009. Browsers were rapidly

changing during this time period; some changes weredirectly motivated by published user studies. Notably,passive indicators are no longer considered primary secu-rity tools, and phishing toolbars have been replaced withbrowser-provided, full-page interstitial warnings. As a re-sult, studies of passive indicators and phishing toolbars nolonger represent the state of modern browser technology.

Two studies tested an older version of the Mozilla Fire-fox SSL warning, in which the warning was a modal(instead of full-page) dialog. Dhamija et al. observeda 68% clickthrough rate, and Sunshine et al. recordedclickthrough rates of 90%-95% depending on the type ofpage [11, 30]. The change in warning design could beresponsible for our lower observed clickthrough rates.

Ecological Invalidity. Sunshine et al. and Sotirakopouloset al. recorded 55%-60% and 80% clickthrough rates, re-spectively, for a slightly outdated version of the MozillaFirefox SSL warning [29, 30]. They evaluated the Fire-fox 3 and 3.5 warnings, which had the same layout andappearance as the current (Firefox 4+) warning but withdifferent wording. Its possible that changes in word-ing caused clickthrough rates to drop from 55%-80% to33.0%. However, during an exit survey, 46% of Soti-rakopouloss subjects said they clicked through the warn-ing because they either felt safe in the laboratory envi-ronment or wanted to complete the task [29]. Since theirstudy methodology was intentionally similar to the Sun-shine study, Sotirakopoulos et al. concluded that bothstudies suffered from biases that raised their clickthroughrates [29]. We therefore attribute some of the discrepancybetween our field study data and these two laboratorystudies to the difficulty of establishing ecological validityin a laboratory environment.

In light of this, we recommend a renewed emphasison field techniques for running and confirming user stud-ies of warnings. Although we used in-browser telemetry,there are other ways of obtaining field data. For exam-ple, experience sampling is a field study methodologythat asks participants to periodically answer questionsabout a topic [2, 6, 9, 27]. Researchers could install abrowser extension on participants computers to observetheir responses to normally occurring warnings and dis-play a survey after each warning. This technique allowsresearchers to collect data about participants emotions,comprehension, and demographics. Participants may be-come more cautious or attentive to warnings if the pur-pose of the study is apparent, so researchers could obscurethe purpose by surveying subjects about other browsertopics. Network-based field measurements also providean alternative methodology with high ecological validity.A network monitor could maintain its own copy of theSafe Browsing list and identify users who click throughwarnings. If the monitor can associate network flows

with specific demographics (e.g., students), it can helpunderstand the impact of these factors on user behavior.Similar studies could help understand SSL clickthroughrates; recent work addressed how to reproduce certificatevalidation at the network monitor [1].

7.3 DemographicsWe found that clickthrough rates differ by operating sys-tem and browser channel. Our findings suggest that highertechnical skill (as indicated by use of Linux and pre-release channels) may predispose users to click throughsome types of warnings. We recommend further inves-tigation of user demographics and their impact on userbehavior. Large-scale demographic studies might uncoveradditional demographic factors that we were unable tostudy with our methodology. If so, can warning designaddress and overcome those demographic differences?

Technically advanced users might feel more confidentin the security of their computers, be more curious aboutblocked websites, or feel patronized by warnings. Studiesof these users could help improve their warning responses.

7.4 Number of ClicksOur data suggests that the amount of effort (i.e., numberof clicks) required to bypass a warning does not alwayshave a large impact on user behavior. To bypass GoogleChromes malware and phishing warnings, the user mustclick twice: once on a small Advanced link, and thenagain to proceed. Despite the hidden button, users clickthrough Google Chromes malware/phishing warning at ahigher rate than Mozilla Firefoxs simpler warning. Fur-thermore, 84% of users who open Mozilla Firefoxs AddException dialog proceed through it.

We find this result surprising. Common wisdom ine-commerce holds that extra clicks decrease clickthroughrates (hence, one-click shopping) [12, 31]. GoogleChromes warning designers introduced the extra step inthe malware/phishing warning because they expected itto serve as a strong deterrent. One possible explanation isthat users make a single cognitive decision when facedwith a warning. The decision might be based on the URL,warning appearance, or warning message. Once the userhas decided to proceed, additional clicks or informationis unlikely to change his or her decision.

Our data suggests that browser-warning designersshould not rely on extra clicks to deter users. However,we did not explicitly design our study to examine theeffects of multiple clicks. Future studies on multi-clickwarnings could shed light on user decision models andimpact security warning design. It is possible that extraclicks do not serve as a deterrent until they reach somethreshold of difficulty.

7.5 Warning Fatigue

We observed behavior that is consistent with the theory ofwarning fatigue. In Google Chrome, users click throughthe most common SSL error faster and more frequentlythan other errors. Our findings support recent literaturethat has modeled user attention to security warnings asa finite resource [4] and proposed warning mechanismsbased on this constraint [14].

Based on this finding, we echo the recommendationthat security practitioners should limit the number of warn-ings that users encounter. Designers of new warningmechanisms should always perform an analysis of thenumber of times the system is projected to raise a warn-ing, and security practitioners should consider the effectsthat warning architectures have on warning fatigue.

7.6 More Information

Users rarely click on the explanatory links such as MoreInformation or Learn More (Section 5.2.4). Designerswho utilize such links should ensure that they do not hidea detail that is important to the decision-making process.

Mozilla Firefox places information about SSL errorsunder Technical Details and in the Add Exceptiondialog instead of the primary warning. Thus, the errortype has little impact on clickthrough rates. In contrast,Google Chrome places error details in the main text ofits SSL warning, and the error has a large effect on userbehavior. It is possible that moving this information intoMozilla Firefoxs primary warning could reduce theirclickthrough rates even further for some errors.

8 Conclusion

We performed a field study with Google Chrome andMozilla Firefoxs telemetry platforms, allowing us to col-lect data on 25,405,944 warning impressions. We findthat browser security warnings can be successful: usersclicked through fewer than a quarter of both browsersmalware and phishing warnings and a third of MozillaFirefoxs SSL warnings. We also find clickthrough ratesas high as 70.2% for Google Chrome SSL warnings, in-dicating that the user experience of a warning can have atremendous impact on user behavior. However, warningeffectiveness varies between demographic groups. Ourfindings motivate more work on browser security warn-ings, with particular attention paid to demographics. AtGoogle, we have begun experimenting with new warningdesigns to further improve our warnings.

Acknowledgements

We thank the participants in Google and Mozillas teleme-try programs for providing us with valuable insight intoour warnings. At Google, we would like to thank MattMueller for setting up the malware and phishing measure-ments, Adam Langley for making suggestions about howto implement SSL measurements, and many others forproviding insightful feedback. At Mozilla, we would liketo thank Sid Stamm for his mentorship and help collectingtelemetry data, Dan Veditz for gathering data from Fire-fox 23, Brian Smith for providing information about thetelemetry mechanisms, and the Mozilla contributors whoreviewed our code and helped land this telemetry [22].We also thank David Wagner, Vern Paxson, Serge Egel-man, Stuart Schechter, and the anonymous reviewers forproviding feedback on drafts of the paper.

References[1] AKHAWE, D., AMANN, B., VALLENTIN, M., AND SOMMER,

R. Heres My Cert, So Trust Me, Maybe? Understanding TLSErrors on the Web. In Proceedings of the 2013 World Wide WebConference (2013).

[2] BEN ABDESSLEM, F., PARRIS, I., AND HENDERSON, T. MobileExperience Sampling: Reaching the Parts of Facebook OtherMethods Cannot Reach. In Privacy and Usability Methods Pow-wow (2010).

[3] BIDDLE, R., VAN OORSCHOT, P. C., PATRICK, A. S., SOBEY,J., AND WHALEN, T. Browser interfaces and extended validationSSL certificates: an empirical study. In Proceedings of the ACMWorkshop on Cloud Computing Security (2009).

[4] BOHME, R., AND GROSSKLAGS, J. The Security Cost of CheapUser Interaction. In Proceedings of the New Security ParadigmsWorkshop (NSPW) (2011).

[5] BRAVO-LILLO, C., CRANOR, L. F., DOWNS, J. S., AND KO-MANDURI, S. Bridging the Gap in Computer Security Warnings:A Mental Model Approach. In IEEE Security and Privacy (March2011), vol. 9.

[6] CHRISTENSEN, T., BARRETT, L., BLISS-MOREAU, E., LEBO,K., AND KASCHUB, C. A Practical Guide to Experience-Sampling Procedures. In Journal of Happiness Studies (2003),vol. 4.

[7] Google Chrome Privacy Notice. http://www.google.com/chrome/intl/en/privacy.html.

[8] CHROMIUM AUTHORS. HSTS Preload and Certificate PinningList. https://src.chromium.org/viewvc/chrome/trunk/src/net/base/transport_security_state_static.json.

[9] CONSOLVO, S., AND WALKER, M. Using the Experience Sam-pling Method to Evaluate Ubicomp Applications. In PervasiveComputing (2003).

[10] Convergence. http://www.convergence.io.

[11] DHAMIJA, R., TYGAR, J. D., AND HEARST, M. Why phishingworks. In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems (2006).

[12] DUTTA, R., JARVENPAA, S., AND TOMAK, K. Impact of Feed-back and Usability of Online Payment Processes on ConsumerDecision Making. In Proceedings of the International Conferenceon Information Systems (2003).

[13] EGELMAN, S., CRANOR, L. F., AND HONG, J. Youve BeenWarned: An Empirical Study of the Effectiveness of Web BrowserPhishing Warnings. In Proceedings of the ACM CHI Conferenceon Human Factors in Computing Systems (2008).

[14] FELT, A. P., EGELMAN, S., FINIFTER, M., AKHAWE, D., ANDWAGNER, D. How to ask for permission. In Proceedings of theUSENIX Conference on Hot Topics in Security (HotSec) (2012).

[15] FRIEDMAN, B., HURLEY, D., HOWE, D. C., FELTEN, E., ANDNISSENBAUM, H. Users Conceptions of Web Security: A Com-parative Study. In CHI Extended Abstracts on Human Factors inComputing Systems (2002).

[16] HABER, J. Smartscreen application reputa-tion in ie9, May 2011. http://blogs.msdn.com/b/ie/archive/2011/05/17/smartscreen-174-application-reputation-in-ie9.aspx.

[17] HERLEY, C. The plight of the targeted attacker in a world of scale.In Proceedings of the Workshop on the Economics of InformationSecurity (WEIS) (2010).

[18] HOLZ, R., BRAUN, L., KAMMENHUBER, N., AND CARLE, G.The ssl landscape: a thorough analysis of the x.509 pki usingactive and passive measurements. In Proceedings of the ACMSIGCOMM Internet Measurement Conference (IMC) (2011).

[19] JACKSON, C., SIMON, D. R., TAN, D. S., AND BARTH, A. Anevaluation of extended validation and picture-in-picture phishingattacks. In Proceedings of the Workshop on Usable Security(USEC) (2007).

[20] LANGLEY, A. SSL Interstitial Bypass Rates, February2012. http://www.imperialviolet.org/2012/07/20/sslbypassrates.html.

[21] MCGRAW, G., FELTEN, E., AND MACMICHAEL, R. SecuringJava: getting down to business with mobile code. Wiley ComputerPub., 1999.

[22] MOZILLA BUGZILLA. Bug 767676: Implement Security UITelemetry. https://bugzil.la/767676.

[23] Mozilla firefox privacy policy. http://www.mozilla.org/en-US/legal/privacy/firefox.html#telemetry.

[24] NETCRAFT. Phishing on sites using ssl cer-tificates, August 2012. http://news.netcraft.com/archives/2012/08/22/phishing-on-sites-using-ssl-certificates.html.

[25] PROVOS, N. Safe Browsing - Protecting Web Users for 5Years and Counting. Google Online Security Blog. http://googleonlinesecurity.blogspot.com/2012/06/safe-browsing-protecting-web-users-for.html, June 2012.

[26] SCHECHTER, S. E., DHAMIJA, R., OZMENT, A., AND FISCHER,I. The Emperors New Security Indicators. In Proceedings of theIEEE Symposium on Security and Privacy (2007).

[27] SCOLLON, C. N., KIM-PRIETO, C., AND DIENER, E. Experi-ence Sampling: Promises and Pitfalls, Strengths and Weaknesses.In Journal of Happiness Studies (2003), vol. 4.

[28] SOBEY, J., BIDDLE, R., VAN OORSCHOT, P., AND PATRICK,A. S. Exploring user reactions to new browser cues for extendedvalidation certificates. In Proceedings of the European Symposiumon Research in Computer Security (2008).

[29] SOTIRAKOPOULOS, A., HAWKEY, K., AND BEZNOSOV, K. Onthe Challenges in Usable Security Lab Studies: Lessons Learnedfrom Replicating a Study on SSL Warnings. In Proceedings of theSymposium on Usable Privacy and Security (2011).

[30] SUNSHINE, J., EGELMAN, S., ALMUHIMEDI, H., ATRI, N.,AND CRANOR, L. F. Crying Wolf: An Empirical Study of SSLWarning Effectiveness. In Proceedings of the USENIX SecuritySymposium (2009).

[31] TILSON, R., DONG, J., MARTIN, S., AND KIEKE, E. Factorsand Principles Affecting the Usability of Four E-commerce Sites.In Our Global Community Conference Proceedings (1998).

[32] WENDLANDT, D., ANDERSEN, D. G., AND PERRIG, A. Perspec-tives: Improving SSH-style Host Authentication with Multi-PathProbing. In USENIX Annual Technical Conference (2008).

[33] WHALEN, T., AND INKPEN, K. M. Gathering evidence: Useof visual security cues in web browsers. In Proceedings of theGraphics Interface Conference (2005).

[34] WU, M., MILLER, R. C., AND GARFINKEL, S. L. Do SecurityToolbars Actually Prevent Phishing Attacks? In Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems(2006).

A Sample Sizes

Malware Phish- SSL Adding Exception

Release 1,968,707 89,948 NC 1,805,928Beta 74,782 3,058 10,976 66,694Dev 61,588 2,759 15,560 53,001Nightly 58,789 4,239 18,617 64,725

Table 7: Warning impression sample sizes for Mozilla Firefoxwarnings, by channel, for all operating systems.

Malware Phish- SSL Adding Exception

Mac 71,371 3,951 534 154,129Win 1,892,285 85,598 10384 1,634,193Linux 1,750 112 58 17,606

Table 8: Warning impression sample sizes for Mozilla Firefoxwarnings, by operating system. The malware, phishing, and theAdd Exception samples are from the release channel, whereasthe SSL samples are from the beta channel. The frame issuedoes not affect statistics that pertain only to the Add Exceptiondialog.

Malware Phishing SSLStable 5,946,057 381,027 16,363,048Beta 44,742 3,525 232,676Dev 14,022 1,186 66,922Canary 35,261 612 42,020

Table 9: Warning impression sample sizes for Google Chromewarnings, by channel, for all operating systems.

In Google Chrome, we recorded 6,040,082 malwarewarning impressions, 386,350 phishing warning impres-sions, and 16,704,666 SSL warning impressions. In

Malware Phishing SSLMac 598,680 20,623 947,971Windows 9,775,104 333,522 13,399,820Linux 15,456 577 515,319Android NC NC 1,499,938

Table 10: Warning impression sample sizes for Google Chromewarnings, by operating system, for the stable channel.

Mozilla Firefox, we recorded 2,163,866 malware warningimpressions, 100,004 phishing warning impressions, and45,153 SSL warning impressions. Tables 7, 8, 9, and 10further separate the sample sizes based on OS and releasechannel.

IntroductionBackgroundMalware and Phishing WarningsSSL WarningsBrowser Release Channels

Prior Laboratory StudiesSSL WarningsPhishing WarningsMalware Download Warnings

MethodologyMeasuring Clickthrough RatesMeasuring Time Spent on WarningsEthicsData CollectionMethod Limitations

Clickthrough RatesMalware and Phishing WarningsMalware Rates by DateMalware/Phishing Rates by Warning TypeMalware/Phishing Rates by DemographicsMalware/Phishing Rates by Browser

SSL WarningsSSL Rates by DemographicSSL Rates by BrowserSSL Rates by Certificate Error TypeAdditional SSL Metrics

Time Spent On SSL WarningsTime by OutcomeTime by Error Type

ImplicationsWarning EffectivenessClickthrough RatesUser Attention

Comparison with Prior ResearchDemographicsNumber of ClicksWarning Fatigue``More Information''

ConclusionSample Sizes