Heartache and HeartbleedAn inside look at the aftermath of Heartbleed

31c3

Nick Sullivan @grittygrease

CloudFlare Reverse Proxy

2

CloudFlare’s Global Network

3

Application Layer• DNS (TCP & UDP port 53)

• HTTP (TCP port 80)

• HTTPS (TCP port 443) - powered by OpenSSL

• Every machine can serve every site

4

Customers

5

Customers

6

It started with a tweet?

7

- Russell Brandom (The Verge)

More like• Do you use TLS heartbeats?

• What is a TLS heartbeat again?

• They’re stupid, you probably don’t need them. Consider turning them off.

• Ok, I’ll compile OpenSSL with -DOPENSSL_NO_HEARTBEATS

8

Then it happened• April 7, 10:27 PDT — OpenSSL publishes advisory

• OpenSSL notification hit #1 on Hacker News

• CloudFlare releases standard “Customer sites are patched” blog post

9

Then it REALLY happened

10

11

HEARTBLEED

Mass-media• Codenomicon launches heartbleed.com with logo

• Heartbleed hits the mainstream press

• #heartbleedvirus trending on Twitter

• My mom calls me

12

Things to do

13

Heartbleed Scanner

14

• Filippo Valsorda’s tool in Go

• Sends a benign heartbeat (~100 bytes)

• Hosted on AWS

15

April 8th (requests/minute)

16

April 8th to April 21

17

203,190,914 testsTotal:

in the first 14 days

% of hosts vulnerable

Meanwhile, at CloudFlare…• Log every heartbeat with a

mismatched length

• Don’t look at data until 31c3

19

Logs from April 9th

20

Malformed Heartbeats Message Size

69% 16384

20% 121

2% 0

8% All other

ssltest.py?

filippo.io

Logs from April 14-16

21

Malformed Heartbeats Message Size

66% 16384

22% 69-131

5% 0

7% All other

ssltest.py?

filippo.ioIP range

1% of all scans

Logs from April 14-16

22

ssltest.py

filippo.ioIP range

Why is Heartbleed so dangerous?

23

• One request gets attacker server data

• Typically not logged — doesn’t leave a trace

• 1.5 million CloudFlare sites share memory

• Login session cookies

• SSL/TLS private keys(???)

24

What does the code say?

25

• Key allocated when process starts

• Copies of keys made at computation time

• OpenSSL bignum library clears allocated memory

• So on a single-threaded server, keys should be safe, right?

The CloudFlare Heartbleed challenge• Let’s crowdsource an answer!

• Standard nginx on digital ocean with vulnerable OpenSSL

• Proof of private key by signing individualized message

26

27

Trolling

Challenge Solved

28

29

Challenge Solved

30

1. Fedor Indutny (@indutny) Developer

2. Ilkka Mattila, Information Security Adviser

3. Rubin Xu (@xurubin), Security PhD Student

4. Ben Murphy (@benmmurphy), Security Researcher

5. Steve Hunter (@nonaxiomatic)

6. Xavier Martin (@xav), Security Researcher

7. no name given

8. Jeremi Gosney (@jmgosney), CEO, Stricture Group

9. Michele Guerini Rocco (@Rnhmjoj), Student

10.David Gervais (@davidgervais), Software Engineer

11.Christian Bürgi (@buergich)

12.Daniel Burkard (@hiptomcat)

• Results: solved in under 10 hours

• Private keys are vulnerable

31

How it was solved• Part of the the private key was on the heap. But why?

• There was a second bug in OpenSSL

32

Second OpenSSL bug

33

• Computation uses temporary variables

• Private key can be derived from them

• Some temporary variables were not wiped

Cleaning up the mess

34

How it was solved - RSA basics• Two prime numbers P & Q

• Public key, including P x Q

• Finding P or Q can get you the private key

35

How it was solved• Take every 128byte block

• Attempt to divide into public RSA key

• Coppersmith’s attack (only requires partial prime factor)

36

37“Revocation”

Revoking 100,000 SSL certificates in 24 hours

38

Revoking 100,000 SSL certificates in 24 hours

39

How revocation works

CRL OCSP

CRLSets40

How revocation works

CRL OCSP

CRLSets41

Revoking 100,000 SSL certificates in 24 hours• GlobalSign CRL grew from 22KB to 4.7MB

• 30Gbps + 100Gbps waves every three hours

42

How revocation works

CRL OCSP

CRLSets43

OCSP is broken• OCSP hard fail breaks captive portals

• Soft fail can be circumvented via network manipulation

• Chrome does not check OCSP

44

How revocation works

CRL OCSP

CRLSets45

CRLSets are broken• Single vendor control

• Only EV certs

• Updates when browser is updated

• None of 100,000+ certs were in CRLSets

• cloudflarechallenge.com was added manually

46

Most efficient revocation code everChromium Issue 267913003

47

How revocation works

CRL OCSP

CRLSets48

Revocation solutions• Shorter certificate expiration periods?

• OCSP Must-staple?

• Certificate Transparency?

49

Things we did

50

Conclusions• Disclosure in open source are hard

• Many “attacks” were scans

• Crowdsourcing was effective

• Revocation needs a solution

51

Heartache and HeartbleedAn inside look at the aftermath of Heartbleed