Breach-zilla: Lessons Learned from Large-Scale Breaches · • Reg keys, services, file settings, etc. • Then, run these scripts daily or weekly – AUTOMATE! Command Line Kung

1

Breach-Zilla ©2011, Ed Skoudis 1

Breach-zilla: Lessons Learned from Large-Scale Breaches

Ed Skoudis v4Q11r


$ cut -f5 -d: /etc/passwd | grep -i skoudis

•  Ed Skoudis •  Started infosec career at Bellcore in 1996 working for phone

companies… eventually got into… –  Pen tests –  Incident response –  Digital forensics

•  SANS Instructor –  Author of classes on Incident Handling (SANS 504), Network Penetration

Testing (560), Windows Command Line (531), and Metasploit (580)

•  InGuardians Co-Founder… Infosec research and consulting •  Author -- Counter Hack Reloaded & Malware - Fighting Malicious Code •  Expert witness on over 100 large-scale breach cases since 2002

2


Purpose

•  I’d like to discuss the trends I’ve seen in breach cases in the past 12 months –  And focus on specific lessons we can learn for

defending our environments better

•  I’d also like to discuss some of the struggles I’m seeing associated with large-scale breach investigations –  In a frank and open fashion

•  And, I plan to leave some time for Q&A at the end


Outline

•  Attack techniques used most often in today’s breaches

•  Issues from breach cases and lessons learned

•  Suggestions for operationalizing intrusion and log analysis

•  Conclusions

3


Common Bad Guy Techniques in Recent Cases

•  Proliferation of initial infection vectors –  SQLi –  Wireless –  Targeted phishing… often as prelude to… –  <blink> Client-side exploitation </blink> –  P2P leakage –  Infected home machine brings attacker in (mobile laptop or VPN)

•  Merciless pivoting –  Flat, unsegmented networks are easy pickin’s for the bad guys –  But, even segmented networks are subject to attack through pivoting… attackers are

getting very clever about pivoting

•  Reverse shell / “phone home” malware –  Often only to single IP address, making it easy to block… this will change

•  DNS tunnel for command and control –  Great theory for years… Now it’s being used! Internal compromised systems don’t

need external access; only the ability to resolve names on the Internet


•  Pass the hash attacks are very widespread –  Grab Windows hashes from one machine and use them to spread throughout

domain without ever knowing the actual password –  Didja see Hernan Ochoa’s new version of Win Credentials Editor (1.2) with pass-

the-ticket for MS Kerberos? (http://www.ampliasecurity.com/research) –  Windows authentication token stealing, especially for domain admin privs

•  Memory scraping –  Bypasses network and file system encryption –  End-to-end encryption is NOT a panacea!

•  Local privilege escalation –  Especially when combined with client-side exploitation

•  Use of sysadmin tools for attack, such as Microsoft’s psexec

–  Remember that it leaves behind a psexec service we can look for •  Some use of custom malware, but often intermixed with common (AV-

detectable) malware

Additional Common Techniques Used in Breaches

4


Outline




•  Conclusions


How Are Breaches Typically Discovered?

•  Most large-scale breach cases are identified first by the card issuers because of fraud with a recent common point of purchase

•  In rare instances, breaches are discovered by the organization itself, but only when the bad guy becomes a hog –  Consuming CPU and/or bandwidth making systems sluggish –  That’s really sad

•  But, in the vast majority of cases, the bad guy did generate some events that the organization could have noticed –  Numerous SQL injection attempts followed by success –  An AV alert, which is auto-quarantined and then ignored –  Widespread intranet scanning –  Admin logon to old account or at unusual time

•  LESSON: Sweat the small stuff, especially AV alerts on a server •  Also, reconfig Win machines to record logon failure and success

5


Did the Organization Have an Idea of the Vulns in Advance?

•  In many cases, yes •  Internal or external auditors and vulnerability assessment

personnel brought issues up in advance •  Organizations didn’t act on the issues •  LESSON: Document the business decision associated with

each audit / vuln assessment finding –  The decision might be to implement another compensating

control, or even to accept the risk –  What’s important here is that there is a paper trail showing that a

decision was made and recorded –  Such documentation leads to better decisions –  It will also help the organization show that it is exercising its due

diligence if a later breach does occur


How Long was the Organization Under Siege?

•  Quite often, attackers are inside a target network for a considerable amount of time –  Usually 6 to 14 months! –  Attackers go unnoticed for that time, as they hunt for the big load

of PII, or dribble it out slowly over long periods of time

•  LESSON: Successful attackers often don’t do a smash and grab, but instead spend a long time inside an organization

•  Their access to the target environment has immense value and takes time, and they won’t give up that asset easily

•  When you discover them in your environment, look for their widespread tracks, and watch for their return

6


What About Re-Infection? •  Many orgs that actually discover a breach block the attacker… •  …but fail to notice the attacker’s return

–  Even though the attacker uses very similar techniques and creates nearly identical artifacts

•  LESSON: After eradication, create scripts that automatically check for attacker’s return: –  Look for directory locations & file names that bad guy created –  Look for software installed by bad guy to return –  Look for changes in configuration that the bad guy made

•  Reg keys, services, file settings, etc.

•  Then, run these scripts daily or weekly – AUTOMATE! Command Line Kung Fu, anyone?


Outline




•  Conclusions

7


•  Most of us know that our organizations should review logs proactively on a regular basis –  But, most organizations don’t –  In many organizations, logs are only consulted after breach is discovered

•  In the breach cases I’ve worked, sometimes there are very good logs –  But no one ever looks at them proactively… –  …despite a written policy and practices that require them to do so

•  There are all kinds of excuses –  “We have a SIM/SEM solution. This is all automated, see!” –  “We don’t have a fancy log infrastructure / correlation tool / SIEM, so we can’t

be expected to actually, you know, look at this stuff.” –  “We’ve got DLP. It won’t let our info leak, see!” –  “We’re too busy!” –  “Our log files are way too big for us to actually look at them!”

•  “Looking for a needle in the haystack is a waste of time!”

•  Bottom line: Routine log analysis is not part of most technical organizations’ culture

Operationalizing Log Analysis


Integrating Log Analysis into the Culture

•  An approach I’ve seen work quite well is to schedule quarterly brown-bag log analysis lunches

•  Reserve a conference room that can hold 6 to 12 people –  Security people and selected system administrators (network admins too!)

•  Have everyone bring logs with them – USB thumb drive with a Gig of logs –  Consider ordering pizza

•  Then, spend an hour with everyone looking through logs and eating lunch –  Splunk, grep –v, findstr /v, and more… low/no cost!

•  If someone sees something unusual or an item that they don’t understand, they can ask the group about it

•  Builds camaraderie and log analysis skills •  And, you might find that needle in the haystack •  Plus, you can now legitimately say that you proactively review logs on a

regular basis

8


Conclusions •  Sensitive data breaches show no signs of letting up

–  Attackers are getting more clever and more lethal than ever

–  More breaches than ever, at a smaller scale of compromised accounts… still messing you up!

•  Thorough incident and log analysis is really helpful, but only if it is done proactively

•  Most organizations need to change their culture regarding log analysis


An Exciting Upcoming Course •  I’ll be teaching my SANS 560 course on Network

Penetration Testing & Ethical Hacking –  Right here, November 7-12

•  The course is designed so you can really understand attacks in depth, find flaws in your organization’s systems, and effect change to improve your security stance

•  It’ll be taught SANS Community-Style –  Smaller classroom sizes (nice!) –  Extra hands-on exercises in bootcamps –  Lower price –  Fun dinners with instructor several nights during the week –  Really a good time and a great value… we’d love to see you there!

9


Q & A

• Any questions? •  Feel free to contact me at

[email protected]

• New site and Pen Test Blog at pen-testing.sans.org/blog – Check it out!

Documents

Breach-zilla: Lessons Learned from Large-Scale Breaches · • Reg keys, services, file settings, etc. • Then, run these scripts daily or weekly – AUTOMATE! Command Line Kung