SSL Certificate Expiration and Howler Monkey's Inception

@royrapoport [email protected]

SSL* Certificate ReportingBayLISA

March 21st, 2013

Friday, March 22, 13

This is the story of how we went from SSL certificates expiring without notice in production to deploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSL certificate expiration as a production-class issue.

mailto:[email protected]



SSL* Certificate ReportingBayLISA

March 21st, 2013


This is the story of how we went from SSL certificates expiring without notice in production to deploying Security Monkey (later renamed Howler Monkey) and permanently eliminating SSL certificate expiration as a production-class issue.




Technology Overview





Technology Overview• SoA, REST, Mostly Java






• Simple overall architecture:






• Simple overall architecture:





Culture Overview


We hire very smart people, give them all the context and situational awareness they want, and set them free. We design our environment, our systems, and our teams to be empowered to make decisions without requiring slow approval processes, cumbersome formal communication, or any other unnecessary friction.




Culture Overview

• Freedom and Responsibility






Culture Overview


• Distributed Operations






Culture Overview


• Distributed Operations

•Get out of the way of Developers






So Certificates ...





So Certificates ...• Dozens of Certificates






• Different kinds of places







• Datacenter/private








• Datacenter/public/LB









• ELBs









• ELBs

• EC2









• ELBs

• EC2

• Source Control









• ELBs

• EC2

• Source Control

• EIPs









• ELBs

• EC2

• Source Control

• EIPs

• Totally Distributed Design





So Certificates ...

• Some Certificates Weren’t[sic]


Some certificates weren’t even SSL certificates -- we have certificates we get from a partner that cannot be accessed via SSL, and for which the answer to the question “when does this expire?” require scraping a web page.




So Certificates ...


(obviously, the ‘standard ways to solve this’ part here is somewhat facetious, but these are, in fact, the standard ways in which most organizations try to deal with keeping up with SSL certificate expirations)




So Certificates ...• SSL Certificates expire







• Millions of people can’t stream








• Hilarity ensues








• Hilarity ensues

• Standard Ways to Solve This








• Hilarity ensues


• Excel worksheets








• Hilarity ensues



• Wiki documents








• Hilarity ensues



• Wiki documents

• Events on public calendars






Let’s Do This ThingCassandra

Certificate


Start with a very simple model -- a Certificate entity, which is really just a combination of name, expiration date, and a series of locations where we can find this. It’d be trivial to feed this thing from my todo list, if I wanted to (but given the state of my todo list, probably a bad idea)





Certificate

ELB


Then start building location-aware spiders -- e.g. this spider that knows how to probe all our ELBs to see if they listen on 443 and gets their certificate if they do.





Certificate

ELB

EC2 Instance


Or this spider that knows how to talk to a specific kind of EC2 instance we have with some certificates.





Certificate

ELB

EC2 Instance

IP Range


etc ...





Certificate

ELB

EC2 Instance

IP Range

Filesystem






Certificate

ELB

EC2 Instance

IP Range

FilesystemDNS






Certificate

ELB

EC2 Instance

IP Range

FilesystemDNS


Once you have all this information, you can easily generate a web page showing certificates, where they are, and when they expire





Certificate

ELB

EC2 Instance

IP Range

FilesystemDNS


And send out emails, too -- once we built the capability for teams to subscribe to emails for a given certificate and specify how many days before expiration they should start getting notified




Since Then


We validated the design by continuing to iterate on it -- recently, when building the DNS spider component, that work took only about 15 minutes to implement. We also expanded subscription capabilities so teams could subscribe to certificate expiration warnings based on certificate name regular expressions.




Since Then

• No Production Emergencies due to SSL certificate expiration






Since Then


• Validated Design






Since Then


• Validated Design

• Better Subscription Capabilities






Soon ...


We should be able to figure out who owns a certificate, most of the time, and alert them directly even if they don’t set up a subscription.




Soon ...• Customized, automated alerting







• Automated renewal








• Telling you a problem is about to happen: Good









• Preventing the problem automatically: Priceless









• Preventing the problem automatically: Priceless

• Open Source






Remember ...





Remember ...

• Be Lazy





Remember ...

• Be Lazy

• Help Others Be Lazy





Remember ...

• Be Lazy


• Computers Are Better Than Humans





Remember ...

• Be Lazy



• For some things





Remember ...

• Be Lazy



• For some things

• Don’t compete on their terms





Questions?


