Splunk for Web Application Security

Kyle Barry

Kyle BarrySecurity Engineering Manager

@allofmywats

The world’s handmade marketplace

30 million members

$895 million in GMS

30% of transactions are international

1.4 billion page views a month

Security at Etsy

sharpwriter.etsy.com

Vulnerabilities

Vulnerabilities

• Cross Site Scripting (XSS)

• Persistent vs. Non-persistent

Vulnerabilities

• Cross Site Scripting (XSS)

• Persistent vs. Non-persistent

• SQL Injection (SQLi)

Vulnerabilities

• Cross Site Scripting (XSS)

• Persistent vs. Non-persistent

• SQL Injection (SQLi)

• Authentication Bypass

Fraud

Fraud

• Account Takeover

Fraud

• Account Takeover

• Phishing

Fraud

• Account Takeover

• Phishing

• Scams

Getting it Done

40+ Daily Deploys

5 Minutes to Production

Config Flags

Continuous Integration

Splunk?

Yes!

Phishing Attackkaroart.etsy.com

An Attack in 4 Parts

Step 1: Alerting

Alerting

• Cron Job?

Alerting

• Cron Job?

• Offline Processing?

Alerting

• Cron Job?

• Offline Processing?

• Real Time?

Alerting

• Cron Job?

• Offline Processing?

• Real Time?

• Splunk

High Number of Logins from a Single IP

Alerting

• Info Log

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info][login] status="success" user="kbarry" ip="123.321.123.321"

• Splunk Search

source="/data/current/web/info.log" action="login" | transaction ip | sort -eventcount | table ip eventcount

Step 2: Damage Control

Damage Control

• Find Affected Accounts

• The Old Way*

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' logfile.log

• Splunk Search

source="/data/current/web/info.log" action="login" | transaction ip | sort -eventcount | table ip eventcount

* : The old way courtesy of http://serverfault.com/questions/11028

Step 3: Look for Trends

Step 4: Clean Up

Clean Up

• Find Affected Accounts

• The Old Way*

awk -F'[ "]+' '$7 == "/" { ipcount[$1]++ } END { for (i in ipcount) { printf "%15s - %d\n", i, ipcount[i] } }' logfile.log

• Splunk Search

source="/data/current/web/info.log" action="login" | transaction ip | sort -eventcount | table ip eventcount

* : The old way courtesy of http://serverfault.com/questions/11028

Putting it Together

Security Mechanisms

Incident Response

Ad-hoc analysis of a large dataset

Driven by an event or incident

Needs to be fast

Reactive Security

Real-time event monitoring and alerting

Events that trigger immediate response

You always query the same data and you do it often

Proactive Security

Things we do now to protect us later

Actions taken to prevent future compromise

Incident Response

wildlifeprints.etsy.com

Phishing Attack

Scanners

Reactive Security

Alerting

• Failed Logins (Info Log)

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info][login] failed login user:"kbarry" ip:"123.321.123.321"

• Splunk Search

source="/data/current/web/info.log" log_namespace="login" | transaction user | sort -eventcount | table user eventcount

source="/data/current/web/info.log" log_namespace="login" | transaction ip | sort -eventcount | table ip eventcount

Alerting

• XSS Alert

source="/data/syslog/current/web/access.log" | regex request_uri="(%3Cscript%3E|alert\(|onerror)"

• SQLi Alert

source="/data/syslog/current/web/info.log" "database syntax error at"

Proactive Security

wildlifeprints.etsy.com

Goal

Full-site SSL for all Etsy sellers

Full Site SSL

• Which pages are important?

source="/data/syslog/current/web/access.log" is_seller=1 | transaction request_uri | table request_uri is_ssl

• How much traffic?

[insert query here]

Security Dashboards

Splunk Friendly Application

Make Logging Easy

<?

$data = array('user' => 'kbarry', 'ip' => 123.321.123.321);

Logger::info('failed captcha', $data,'login');

?>

Make Splunking Easy

Access Log[Wed Apr 22 16:37:41 2013] "POST /signin HTTP/1.1" 200 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" MF9JqDVpY93VOMreyvI2UC24wRjT

Info Log

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info] [login] failed captcha user:"kbarry" ip:"123.321.123.321"

Global Request ID

Access Log[Wed Apr 22 16:37:41 2013] "POST /signin HTTP/1.1" 200 "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11" MF9JqDVpY93VOMreyvI2UC24wRjT

Info Log

[Wed Apr 22 16:37:41 2013] [MF9JqDVpY93VOMreyvI2UC24wRjT] [info] [login] failed captcha user:"kbarry" ip:"123.321.123.321"

Recap

Recap

• The More You Know (TM)

Recap

• The More You Know (TM)

• Security is a Real Time Problem

Recap

• The More You Know (TM)

• Security is a Real Time Problem

• Instrument your Application Appropriately