Heartache and HeartbleedAn inside look at the aftermath of Heartbleed
31c3
Nick Sullivan @grittygrease
CloudFlare Reverse Proxy
2
CloudFlare’s Global Network
3
Application Layer• DNS (TCP & UDP port 53)
• HTTP (TCP port 80)
• HTTPS (TCP port 443) - powered by OpenSSL
• Every machine can serve every site
4
Customers
5
Customers
6
It started with a tweet?
7
- Russell Brandom (The Verge)
More like• Do you use TLS heartbeats?
• What is a TLS heartbeat again?
• They’re stupid, you probably don’t need them. Consider turning them off.
• Ok, I’ll compile OpenSSL with -DOPENSSL_NO_HEARTBEATS
8
Then it happened• April 7, 10:27 PDT — OpenSSL publishes advisory
• OpenSSL notification hit #1 on Hacker News
• CloudFlare releases standard “Customer sites are patched” blog post
9
Then it REALLY happened
10
11
HEARTBLEED
Mass-media• Codenomicon launches heartbleed.com with logo
• Heartbleed hits the mainstream press
• #heartbleedvirus trending on Twitter
• My mom calls me
12
Things to do
13
Heartbleed Scanner
14
• Filippo Valsorda’s tool in Go
• Sends a benign heartbeat (~100 bytes)
• Hosted on AWS
15
April 8th (requests/minute)
16
April 8th to April 21
17
203,190,914 testsTotal:
in the first 14 days
% of hosts vulnerable
Meanwhile, at CloudFlare…• Log every heartbeat with a
mismatched length
• Don’t look at data until 31c3
19
Logs from April 9th
20
Malformed Heartbeats Message Size
69% 16384
20% 121
2% 0
8% All other
ssltest.py?
filippo.io
Logs from April 14-16
21
Malformed Heartbeats Message Size
66% 16384
22% 69-131
5% 0
7% All other
ssltest.py?
filippo.ioIP range
1% of all scans
Logs from April 14-16
22
ssltest.py
filippo.ioIP range
Why is Heartbleed so dangerous?
23
• One request gets attacker server data
• Typically not logged — doesn’t leave a trace
• 1.5 million CloudFlare sites share memory
• Login session cookies
• SSL/TLS private keys(???)
24
What does the code say?
25
• Key allocated when process starts
• Copies of keys made at computation time
• OpenSSL bignum library clears allocated memory
• So on a single-threaded server, keys should be safe, right?
The CloudFlare Heartbleed challenge• Let’s crowdsource an answer!
• Standard nginx on digital ocean with vulnerable OpenSSL
• Proof of private key by signing individualized message
26
27
Trolling
Challenge Solved
28
29
Challenge Solved
30
1. Fedor Indutny (@indutny) Developer
2. Ilkka Mattila, Information Security Adviser
3. Rubin Xu (@xurubin), Security PhD Student
4. Ben Murphy (@benmmurphy), Security Researcher
5. Steve Hunter (@nonaxiomatic)
6. Xavier Martin (@xav), Security Researcher
7. no name given
8. Jeremi Gosney (@jmgosney), CEO, Stricture Group
9. Michele Guerini Rocco (@Rnhmjoj), Student
10.David Gervais (@davidgervais), Software Engineer
11.Christian Bürgi (@buergich)
12.Daniel Burkard (@hiptomcat)
• Results: solved in under 10 hours
• Private keys are vulnerable
31
How it was solved• Part of the the private key was on the heap. But why?
• There was a second bug in OpenSSL
32
Second OpenSSL bug
33
• Computation uses temporary variables
• Private key can be derived from them
• Some temporary variables were not wiped
Cleaning up the mess
34
How it was solved - RSA basics• Two prime numbers P & Q
• Public key, including P x Q
• Finding P or Q can get you the private key
35
How it was solved• Take every 128byte block
• Attempt to divide into public RSA key
• Coppersmith’s attack (only requires partial prime factor)
36
37“Revocation”
Revoking 100,000 SSL certificates in 24 hours
38
Revoking 100,000 SSL certificates in 24 hours
39
How revocation works
CRL OCSP
CRLSets40
How revocation works
CRL OCSP
CRLSets41
Revoking 100,000 SSL certificates in 24 hours• GlobalSign CRL grew from 22KB to 4.7MB
• 30Gbps + 100Gbps waves every three hours
42
How revocation works
CRL OCSP
CRLSets43
OCSP is broken• OCSP hard fail breaks captive portals
• Soft fail can be circumvented via network manipulation
• Chrome does not check OCSP
44
How revocation works
CRL OCSP
CRLSets45
CRLSets are broken• Single vendor control
• Only EV certs
• Updates when browser is updated
• None of 100,000+ certs were in CRLSets
• cloudflarechallenge.com was added manually
46
Most efficient revocation code everChromium Issue 267913003
47
How revocation works
CRL OCSP
CRLSets48
Revocation solutions• Shorter certificate expiration periods?
• OCSP Must-staple?
• Certificate Transparency?
49
Things we did
50
Conclusions• Disclosure in open source are hard
• Many “attacks” were scans
• Crowdsourcing was effective
• Revocation needs a solution
51
Heartache and HeartbleedAn inside look at the aftermath of Heartbleed