5 Bare Minimum Things a Web startup CTO MUST worry about

Indus Khaitanhttp://khaitan.org

khaitan@gmail.com

Twitter: 1ndus

*Not affiliated to any software vendors mentioned in this preso

… and implement few basic things to have a good night’s sleep!

18 slides

Mar, 2009 Barcamp Bangalore 2

What are these?

1. Security

2. Availability & Monitoring

3. Application Errors

4. Backup

5. Source Control

(in order of decreasing priority)

Mar, 2009 Barcamp Bangalore 3

Security Threats

• Your website taken over

• Your database taken over

• Your server taken over

• (Distributed) Denial of Service

Mar, 2009 Barcamp Bangalore 4

Prevention of Security Threats

• Keep your stack up-to-date. Patch.• Establish security-aware coding practice• Know your Logs!• Install open source packages for

preventive/reactive treatments• Get a hardware firewall (if you are popular and

have money)

… Subscribe to Securityfocus alerts

Mar, 2009 Barcamp Bangalore 5

Simple TODO List for You

• Use logwatch and monitor your logs• Make your Database access local (specific IPs

only)• Secure your sshd

– Password-less login, non-default port, no root login

• Use denyhosts to block dictionary SSH attacks (iptables/netfilter is a good bet, I haven’t tried it)

• Close all ports except SSH, HTTP/HTTPS– Use nmap to see what “hackers” see!

Mar, 2009 Barcamp Bangalore 6

A log snapshot of SSH attack Didn't receive an ident from these IPs: 114.200.199.144: 1 Time(s)

Illegal users from: 114.200.199.144: 6 times alias/password: 1 time office/password: 1 time recruit/password: 1 time sales/password: 1 time samba/password: 1 time staff/password: 1 time Failed logins from: 211.60.15.30: 1 time root/password: 1 time 219.137.24.12: 1 time root/password: 1 time

Mar, 2009 Barcamp Bangalore 7

Availability & Monitoring

• Website, Database, SMTP, DNS were down (now up!)

• Poor site performance – Application, Network, or hosting provider?

• CPU, Disk, IO, Memory, Network Interface

• Server down != website down. Put a load balancer

Mar, 2009 Barcamp Bangalore 8

Monitoring – External sample

Mar, 2009 Barcamp Bangalore 9

Monitoring: InternalSystem Level Monitoring with Nagios

Mar, 2009 Barcamp Bangalore 10

Simple TODO List for You

• Do some basic external monitoring – Zoho does url/5minutes at $4/month...cheap!

• Get Nagios for system monitoring

• Use Load Balancer to prevent single server failure– HTTP, Load Balanced database reads

Mar, 2009 Barcamp Bangalore 11

Application Errors

Bad Codefunction validate($key) {

global $weblog ;

if (empty($key)) {

$errorlog->error( "Error : In function validate site key");

return FALSE;

}else{

return TRUE;

}

}

Leads to this in phperror log[13-Feb-2009 09:41:32] PHP Fatal error: Call to a member function error() on a non-object in

/home/padmin/public_html/util/functions.php on line 4

Mar, 2009 Barcamp Bangalore 12

Application Errors

• Simple WARNINGS/FATALs lead to bigger problems– eg. INSERT failed because of duplicate key

(was always inserting 0 for the parameter!)

• Apache error_log may show wrong configuration

• Database logs may show a crash (and auto-recovery!)

Mar, 2009 Barcamp Bangalore 13

Simple TODO for You

• Use a logger like log4j/log4PHP– Modify the handler to send a real-time email

of a desired error level

• Look for Database Error logs, Apache error logs – They will tell you a story!

• Borrow from Security: Use logwatch package

• Review your own application codebase

Mar, 2009 Barcamp Bangalore 14

Backup

• Backup before disaster strikes

• Database backups– Do a dry run of recovery at least once– Ensure consistent, online backups

• Backup your production directories

Mar, 2009 Barcamp Bangalore 15

Simple TODO For You

• (mysql) Use a slave for a consistent backup. No slave? Then Lock the master before dumping

• Take a backup tar of production– Preferably backed up every week, and just

before a deployment and just after a deployment

• Use S3 to store the files remotely

Mar, 2009 Barcamp Bangalore 16

Source Control: Simple TODO For You

• Use SVN– Use hosted… DevGuard..$7/month..cheap!

• Few Developers? Can’t do Linux? No money? Use a local SVN server on Windows. Woorrks! But back-it-up!!!

• Have a prod. deployment strategy– From SVN, DON’T deploy directly on Prod.,

use a separate instance and then scp/rsync over

Mar, 2009 Barcamp Bangalore 17

Summary

• Know Your Logs!

• Be Security aware– Lock your SSH. Close Open Ports

• Do some basic external monitoring

• Backup your Database & prod directory onto a remote location

• Use SVN

Mar, 2009 Barcamp Bangalore 18

Sample Advanced Topics & Thanks!

• Incremental backups, snapshots

• Monitoring Apache Processes, Apache IO, Database connections, Load, Query/sec

• Using SSH Tunneling

• Virtual Private & Public LANs

• VPN