Confused JohnnyWHEN AUTOMATIC ENCRYPTIONLEADS TO CONFUSION AND MISTAKESScott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent SeamonsInternet Security Research LabComputer Science DepartmentBrigham Young University

Confused Johnnyo E-mail encryption for the masses

o We developed a system maximizing usability◦ Made everything transparent

o Johnny became confused

o Designed another system with manual encryption◦ This helped Johnny gain clarity

Encrypted E-mailo Exists, but largely goes unused

o S/MIME, PGP◦ Tools available

o “Why Johnny can't encrypt: A usability evaluation of PGP 5.0”◦ Whitten and Tygar, 8th USENIX Security Symposium (1999)◦ Later research confirmed findings

o What can be done?

Usability Issueso Users resist change

◦ Users are using webmail◦ If security is difficult users will forgo it

o Key management is confusing◦ Hierarchical, web-of-trust◦ Recipient must already have key◦ Chicken and egg problem

o Cryptography is complicated◦ Unclear which properties are provided◦ Unclear which properties are needed

Private Webmail (Pwm)o Pronounced “Poem”

o Adds end-to-end encryption to existing webmail systems◦ Gmail, Hotmail, Yahoo! Mail◦ Runs on all modern browsers

o Designed to maximize usability

o Provide good-enough security◦ Improvement for those already sending sensitive e-mail

Users Resist Changeo Security overlays

◦ Integrates tightly with existing webmail systems◦ Users do not need to learn yet-another-system

o Tightly integrates with existing systems◦ Replaces small portions of the interface◦ Displayed using iFrames

o Functionally transparent◦ Low barrier to adoption

o Visually distinctive◦ Easy to identify

Usability Fixeso Users resist change

◦ Focus on bootstrapping first-time users◦ Helpful instructions in e-mail◦ Bookmarklet-based installation

o Key management is confusing◦ Key escrow based on IBE◦ Simple Authentication for the Web (EBIA)◦ No user interaction required

o Cryptography is complicated◦ Encryption is automatically handled by Pwm◦ Users never interact with ciphertext

Pwm: Walkthrough

Pwm: Walkthrough

Pwm: Walkthrough

Pwm: Walkthrough

Pwm: Walkthrough

Pwm User Studieso Two studies

o First study measured usability of Pwm◦ Also evaluated bookmarklets for use during installation

o Second study compared Pwm to Voltage Secure Mail Cloud◦ Voltage Secure Mail Cloud is an existing depot-based secure email system◦ Pwm was run using a browser extension

o Evaluation◦ Pre- and post-survey questionnaire◦ Monitored participants actions for unrecognized mistakes◦ Post-survey interviews

System Usability Scaleo Brook (1996)

o Ten questions◦ Alternate negative and positive◦ Give a single number for usability

o Bangor compared scores for hundreds of systems

76

71

63

SUS Score Comparison

Success?o Results are very promising

◦ Very positive reception◦ Users indicated they wanted to begin using it

o Not without problems

o Small number sent e-mail without encryption

o Participants were confused about security◦ Wanted to see more details◦ Unsure of who could read e-mails

Where to go from here?o Simple solutions was to fix UI issues

o One author (Nathan Kim) had a different idea◦ Manual encryption◦ Decoupled interface

o Mocked up these ideas◦ Message Protector (MP)◦ Simple Interface◦ Direct handling of ciphertext◦ Implied key management

MP: Walkthrough

MP: Walkthrough

MP: Walkthrough

MP: Walkthrough

First MP User Studyo Evaluated MP using SUS

o Compared against Encipher.it◦ Bookmarklet-based encryption system◦ Works in Gmail and Facebook

o Evaluation◦ Pre- and post-survey questionnaire◦ Monitored participants actions for unrecognized mistakes◦ Post-survey interviews◦ The system usability scale

o Evaluated comprehension◦ Survey included questions about comprehension◦ How to use the system◦ Who could read messages

61

72

Second MP User Studyo Surprising usability results

◦ Participants had a positive reaction to seeing ciphertext◦ Similar SUS score to MP

o Ran a second study comparing MP to Pwm◦ Modeled after the first MP study

76

74

SUS Score Comparison

Other resultso MP improved users comprehension

◦ Clearly understood how to use system◦ Clearly understood who could read messages

o Usability scores nearly identical to Pwm

o Participants preferred manual encryption of MP

o Participants preferred tight integration of Pwm

Study limitationso MP studies ignore bootstrapping new users

◦ Studies assumed software pre-installed◦ Bootstrapping is a key component of Pwm’s design◦ Not fully representative of overall usability

o Short-term studies

o SUS question unclear◦ “I think that I would like to use this system frequently.”◦ Participants ranked low even when enthusiastic about the system◦ Relevant to security studies

Related Workso “Johnny 2: a user test of key continuity management with s/mime and outlook express.”

◦ Garfinkel and Miller, SOUPS 2005◦ Applied automatic key management to e-mail◦ Allowed great success

o “Helping Johnny 2.0 to encrypt his Facebook conversations.”◦ Fahl, et al., SOUPS 2012◦ Encrypted Facebook communication◦ Explored manual vs. automatic encryption◦ Invisibility security not trusted by users

Conclusiono Pwm was a success

◦ Participants largely succeeded at using encrypted e-mail◦ Participants had high praise for Pwm◦ Succeeding in being easy for new users

o Pwm wasn’t perfect◦ Security was too transparent◦ Caused users to be confused and make mistakes

o Mocked up a system using manual encryption◦ Users enjoyed manual encryption◦ Wished it was tightly integrated with the browser

o A combination of approaches is needed to solve the problem

Conclusion: Future Worko Manual encryption in Pwm

◦ Don’t automatically send encrypted email◦ “Encrypt” button which puts ciphertext in compose window

o Sidebar◦ Browser sidebar allowing for manual encryption◦ Can be used on any site◦ Fallback for when Pwm has an error

o Long-term studies◦ Larger populations◦ Real tasks

Questions?