Confused JohnnyWHEN AUTOMATIC ENCRYPTIONLEADS TO CONFUSION AND MISTAKESScott Ruoti, Nathan Kim, Ben Burgon, Tim van der Horst, Kent SeamonsInternet Security Research LabComputer Science DepartmentBrigham Young University
Confused Johnnyo E-mail encryption for the masses
o We developed a system maximizing usability◦ Made everything transparent
o Johnny became confused
o Designed another system with manual encryption◦ This helped Johnny gain clarity
Encrypted E-mailo Exists, but largely goes unused
o S/MIME, PGP◦ Tools available
o “Why Johnny can't encrypt: A usability evaluation of PGP 5.0”◦ Whitten and Tygar, 8th USENIX Security Symposium (1999)◦ Later research confirmed findings
o What can be done?
Usability Issueso Users resist change
◦ Users are using webmail◦ If security is difficult users will forgo it
o Key management is confusing◦ Hierarchical, web-of-trust◦ Recipient must already have key◦ Chicken and egg problem
o Cryptography is complicated◦ Unclear which properties are provided◦ Unclear which properties are needed
Private Webmail (Pwm)o Pronounced “Poem”
o Adds end-to-end encryption to existing webmail systems◦ Gmail, Hotmail, Yahoo! Mail◦ Runs on all modern browsers
o Designed to maximize usability
o Provide good-enough security◦ Improvement for those already sending sensitive e-mail
Users Resist Changeo Security overlays
◦ Integrates tightly with existing webmail systems◦ Users do not need to learn yet-another-system
o Tightly integrates with existing systems◦ Replaces small portions of the interface◦ Displayed using iFrames
o Functionally transparent◦ Low barrier to adoption
o Visually distinctive◦ Easy to identify
Usability Fixeso Users resist change
◦ Focus on bootstrapping first-time users◦ Helpful instructions in e-mail◦ Bookmarklet-based installation
o Key management is confusing◦ Key escrow based on IBE◦ Simple Authentication for the Web (EBIA)◦ No user interaction required
o Cryptography is complicated◦ Encryption is automatically handled by Pwm◦ Users never interact with ciphertext
Pwm: Walkthrough
Pwm: Walkthrough
Pwm: Walkthrough
Pwm: Walkthrough
Pwm: Walkthrough
Pwm User Studieso Two studies
o First study measured usability of Pwm◦ Also evaluated bookmarklets for use during installation
o Second study compared Pwm to Voltage Secure Mail Cloud◦ Voltage Secure Mail Cloud is an existing depot-based secure email system◦ Pwm was run using a browser extension
o Evaluation◦ Pre- and post-survey questionnaire◦ Monitored participants actions for unrecognized mistakes◦ Post-survey interviews
System Usability Scaleo Brook (1996)
o Ten questions◦ Alternate negative and positive◦ Give a single number for usability
o Bangor compared scores for hundreds of systems
76
71
63
SUS Score Comparison
Success?o Results are very promising
◦ Very positive reception◦ Users indicated they wanted to begin using it
o Not without problems
o Small number sent e-mail without encryption
o Participants were confused about security◦ Wanted to see more details◦ Unsure of who could read e-mails
Where to go from here?o Simple solutions was to fix UI issues
o One author (Nathan Kim) had a different idea◦ Manual encryption◦ Decoupled interface
o Mocked up these ideas◦ Message Protector (MP)◦ Simple Interface◦ Direct handling of ciphertext◦ Implied key management
MP: Walkthrough
MP: Walkthrough
MP: Walkthrough
MP: Walkthrough
First MP User Studyo Evaluated MP using SUS
o Compared against Encipher.it◦ Bookmarklet-based encryption system◦ Works in Gmail and Facebook
o Evaluation◦ Pre- and post-survey questionnaire◦ Monitored participants actions for unrecognized mistakes◦ Post-survey interviews◦ The system usability scale
o Evaluated comprehension◦ Survey included questions about comprehension◦ How to use the system◦ Who could read messages
61
72
Second MP User Studyo Surprising usability results
◦ Participants had a positive reaction to seeing ciphertext◦ Similar SUS score to MP
o Ran a second study comparing MP to Pwm◦ Modeled after the first MP study
76
74
SUS Score Comparison
Other resultso MP improved users comprehension
◦ Clearly understood how to use system◦ Clearly understood who could read messages
o Usability scores nearly identical to Pwm
o Participants preferred manual encryption of MP
o Participants preferred tight integration of Pwm
Study limitationso MP studies ignore bootstrapping new users
◦ Studies assumed software pre-installed◦ Bootstrapping is a key component of Pwm’s design◦ Not fully representative of overall usability
o Short-term studies
o SUS question unclear◦ “I think that I would like to use this system frequently.”◦ Participants ranked low even when enthusiastic about the system◦ Relevant to security studies
Related Workso “Johnny 2: a user test of key continuity management with s/mime and outlook express.”
◦ Garfinkel and Miller, SOUPS 2005◦ Applied automatic key management to e-mail◦ Allowed great success
o “Helping Johnny 2.0 to encrypt his Facebook conversations.”◦ Fahl, et al., SOUPS 2012◦ Encrypted Facebook communication◦ Explored manual vs. automatic encryption◦ Invisibility security not trusted by users
Conclusiono Pwm was a success
◦ Participants largely succeeded at using encrypted e-mail◦ Participants had high praise for Pwm◦ Succeeding in being easy for new users
o Pwm wasn’t perfect◦ Security was too transparent◦ Caused users to be confused and make mistakes
o Mocked up a system using manual encryption◦ Users enjoyed manual encryption◦ Wished it was tightly integrated with the browser
o A combination of approaches is needed to solve the problem
Conclusion: Future Worko Manual encryption in Pwm
◦ Don’t automatically send encrypted email◦ “Encrypt” button which puts ciphertext in compose window
o Sidebar◦ Browser sidebar allowing for manual encryption◦ Can be used on any site◦ Fallback for when Pwm has an error
o Long-term studies◦ Larger populations◦ Real tasks
Questions?
Recommended
