Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Frédéric G. MARAND http://www.osinet.fr/
So you’ve been hacked…now what ?
1
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
Table of Contents
1 Intro : setting the stage 2 Snapshotting 3 Maintaining presence 4 Crisis communication 5 Rebuild, don’t repair 6 Using forensics tools 7 Back online
2
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.1 Some fact checking first
• In this room … • Whose site has been hacked already ?
3
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.1 Some fact checking first
• In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ?
4
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.1 Some fact checking first
• In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ?
5
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.1 Some fact checking first
• In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? • Who actually has a GDPR-compliant business continuity plan ?
6
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.1 Some fact checking first
• In this room … • Whose site has been hacked already ? • Who feels ready to face a hacked server ? • Who actually has a business continuity plan ? • Who actually has a GDPR-compliant business continuity plan ? • Who read node 2365547 ?
7
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 8
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.2 Can you say that again ?
9
I.A.N.A.L. (and you probably ain’t either)
So be sure to get one !
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.3 Whence do I speak ?
• Drupal org member since 2005 (fgm) • Drupal consultant, not a site building agency • Worked on fixing broken(-in) sites since 2008 • Auditing • Fixing technical flaws • Addressing intrusions / exploits
• Mostly Media and Government sites (.fr) • « Provisional member » on the Security Team
10
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.4 Choose your own adventure
• 10:00 The daily scrum has just begun. • 10:01 Phones rings : someone noticed your site has been defaced and
is warning you • 10:02 Twitter and Reddit start buzzing • 10:05 Phones ring all over the place, journalists and CxO types on the
other end, asking about GDPR Art. 33/34 ; your mailbox is filling with warnings • What is your next step ?
11
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
1.5 Get ready
• Notepad 1 : discovery log • all your work steps, to the
tiniest • all your findings /
observations • with timestamps and
numbers
12
• Notepad 2 : remedies ideas • all your ideas for fixing
the breach • all your ideas for further
hardening • cross-refer notepad 1
numbers
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
2.1 Forensic copy : why ?
• Obvious temptation: • restore & resume • Still vulnerable • So you need to diagnose
• Analyzing means modifying • So preserve the « crime scene » • Snapshot everything
13
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
2.2 Snapshots : pull the plug ?
• Prevents interference • Shutdown handlers, SIGPWR • Self-destructing code on
network loss • Easy on VMs, cloud instances • No need to pull the plug
14
• But… • Bare remote servers • Further data loss • Journaled FS • Databases
• Service interruption
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
2.3 Snapshots : everything == ?
• Not just the main DB
• Reverse Proxy logs • Web fronts • DB servers • File servers
• Also…
• External logs (SaaS) • External transactions • FW/IDS/WAF logs
15
The site might be just another attack vector, not the main target
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
3.1 Maintaining presence #1
• Pros • Don’t tip off hackers: you’re
one step ahead until they notice • Keep on generating short-
term value
• Cons • Damage increase • Responsibility • Legal (GDPR) • Financial • Moral
16
As though intrusion had not been detected
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
3.2 Attacker workflows (intermezzo)
Modern: multistep flow
• Break in
• Dig for gold • Implant zombie
• Wait for implant migration to archives
• Activate • Profit
Amateur: Need for Speed
• Use exploit ASAP
• While it lasts • Usually least loss (miners)
Art crime: hide the act
• Valuable content
• Identity data • Close the door
17
(➡ forensics)(➡ restore)
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
3.3 Maintaining presence #2
• Limited static site • Best with prior work • Minimal subset • Possibly taken from RP cache
• Very little load • Can run off RP heads
• Working limited (read-only) site • Alternate infra • Alternate tech
• Updates ? • Content created during
crisis
18
Safe fallback mode
Always easier if planned from the onset. Can help with GDPR Art 41.2b
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
3.4 Maintaining presence #3
• Social networks and status sites • Always there • Also authoritative for audience
• Still needs some preparation • Accounts access • Inclusion in long-term communication
• B2C: Facebook / Instagram pages • B2B: Twitter, status sites
19
When all else fails
http
s://
xeno
mor
ph01
.dev
iant
art.
com
/art
/Dea
dpoo
l-ti
s-bu
t-a-
scra
tch-
4793
3864
0
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
4.1 Communicating : upstream
• Internal stakeholders • DPO if any personal data are involved • CxO level in most cases
• Gag orders ? Protection may exist. But follow the rules. • France: whistleblower protection in Sapin 2 (limited) • Italy (banks): Dec. 385 01/09/93 sect 52bis • US (contractors): Anti-SLAPP • Many countries have similar rules
20
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
4.2 : Communication : C-level
• DPO (start here), Legal counsel • Crisis Management specialists (costs…)
• Law enforcement • National data protection agencies within 72h (GDPR Art. 33, Rec. 85) • EU countries typically have« cybercrime » units like (FR) ANSSI
• Other sites • On same server • On same network • Online business partners ➡ GDPR: Data Processors or 3rd parties ?
21
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
4.3 Communication : privacy
• Often personal data leaks • will happen, or... • unprovable they did not
• Operational constraints • Commerce : PCI/DSS
(12 steps etc) • Health : (US) HIPAA
Subtitle D E2.80.93 • Public image damage control
• GDPR Conflicting rules • 72h delay for sensitive
exploits (Art. 33.1), or justification needed • Police operations need time.
Per GDPR, police prevails (Rec. 86)
22
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
5.1 Rebuild : keep, rollback or ..?
• Restore and restart same ? • Still just as vulnerable, just
more so • Keep and fix ? • lots of time and effort
reviewing • never completely trusted : not
just Drupal
• Throw away ? • Event sites, past lines of biz,
post-M&A... • Can a static version suffice ? • From RP snapshots : recent
content • Best: rebuild from sources +
restore backups
23
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
5.2 Rebuild : restore content
• Needs backups from before the hack • Do you know when it happened ? Hint: « modern workflow » • GFS, continuous incremental, 15 min ? • How much can you lose ?
• FLOSS solutions : Amanda, Bacula, custom… • Unprepared emergency ? • Preproduction, CI builds…
24
(➡ modern workflow)
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
5.3 Rebuild : sources + export
• Easy and reliable, but assumes : • Code-driven development process • Reliable data export system in place • Accessible content exports (e.g. flat files, journals) • Content + assets repositories
• Still need to add the fixes • Delay can be a problem on high-volume sites • Bulk handling, Incremental loading
25
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
5.4 Rebuild : other cases
• Ad hoc « traditional » build process • Longer, less reliable • Too long to be a chance to fix the process
• From scratch • Too long in most (all ?) cases • Do it as a complement after the fix • Not NOW
26
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6 Forensics : switching hats
27
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.1 Forensics : first, think !• How did you discover the hack ? • What did it take to succeed ? • Cast your net wide, think big • « Unlikely » vs « impossible »
• Priority : • Easiest attacks first • OWASP 10 • GIYF : search for notepad 1
patterns
28
« Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. » Arthur Conan Doyle «The sign of the four »
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
• /anything/ may be erased after success • But most of the time, not /everything/ will
• Anything you do leaves its own traces • Work on copies of the snapshots • You can restart from fresh copies anytime
• There maybe more than one exploit • « Cleaner exploits » after Drupalgeddon 1 • « art crime » workflow
6.2 Forensics : keep in mind
29
(➡art crime)
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.3 Forensics : classic flaws
• Code files : • lax permissions • filesystem traversal issues • Remote payload execution by upload
• Nginx without extra hardening • .htaccess won’t do much good
• In-DB PHP • PHP module
• Eval-uated code
30
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.4 Forensics : outer droppings
• Filesystem : • <user>/www-data outside /sites
• www-data/www-data suspicious • « x » bit (0111) on files below docroot
• timestamps • ts(outside sites/*/files) = ts(install) • ts(exploits) > ts(install)
• meld with a fresh build from sources • Also check outside docroot / project root
31
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.5 Forensics : Drupal modules
• Code signing/diffing : • Hacked! Limitations • D7 : md5check, file_integrity
• Finding DB PHP • OSInet QA (github)
• Misc • security_review
32
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.6 Forensics : DB scans
• Quick wins : • users_field_data.email != users_field_data.init (users on D6/D7) • review roles, accounts with admin roles • On corporate sites, users_field_data.email domains • match users accounts with SSO data, directories
• Diff DB snapshot with live • D7: menu_router : file_put_contents, assert • D8: use vendor/bin/drupal debug:router instead of DB
• Altova DatabaseSpy content compare
33
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.7 Forensics : session data
• Sessions should be in persistent storage • Remember when you pulled the plug • Were your sessions in Memcache ? in-memory Redis ?
• Quick checks: • sessions.timestamp vs users_field_data : created / changed / access / login • for intranets : sessions.hostname
34
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.8 Forensics : logs• You use off-site logs, right ? • SaaS : Loggly, Logmatic,
Logsene, Logz.io, Papertrail, Scalyr…. • Homegrown remote ELK • GDPR • Data Processor constraints
(Rec. 81) • Record of processing
activities (Art. 30)
• Still on-site ? Read-write ? • dblog {watchdog} • syslog → redirects chain • mongodb_watchdog • redis_watchdog • GELF/Graylog, Logstash
• Application/WS logs
35
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
6.9 Forensics : sleuth tools• Software • Guidance Software :
Encase • AccessData : Forensics
Toolkit (FTK) • Certified consultants • National cybercrime units
36
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
7.1 Live again : restoring production
• On preprod, recheck notepad 1 findings vs new build • Usually, reset passwords. • On D8, use mass_pwreset • On D7/MySQL: • update users set pass = concat(‘ZZZ', sha(concat(pass, md5(rand()))));
• Prepare the GDPR Art. 34 exploit report • Prepare marketing/social copy • Prepare for the future
37
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
7.2 L8R : future-readiness
38
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
7.3 L8R: be prepared
• Developer education on security • Security Team mailing list • https://twitter.com/drupalsecurity • https://www.drupal.org/security/rss.xml • http://crackingdrupal.com/ ?
• National support • GDPR requires national authorities to
provide education (Art. 57, Rec. 122, 132) • Exists outside GDPR (FR: ANSSI MOOC)
https://secnumacademie.gouv.fr/
39
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
7.4 L8R : disaster prevention
• Security process • Analyse sec. releases to
understand fixes • Look for similar flaw in
custom code • Take part in Drupal core /
contrib to acquire expertise
• Quality process • Systematic peer code reviews • Code-driver maintenance +
dev process • Automatic quality tools in CI • Contrib updates scheduling
40
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0
7.5 Continuous improvement
• « You can’t improve what you don’t measure » • Get time metrics from notepad 1 • Build contingency plan from notepad 2
• Plan for periodic intrusion simulations • GDPR Art. 32.1d: « …the controller and the processor shall
implement […] a process for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures for ensuring the security of the processing »
41
©2018 OSInet - Licensed under Creative Commons CC-BY-SA 4.0 42
Drupal, faster, safer http://www.osinet.fr/
@OSInethttps://www.linkedin.com/company/osinet/
https://goo.gl/X9LEzzRate this talk: