5 Bare Minimum Things a Web startup CTO MUST worry about
Indus Khaitanhttp://khaitan.org
Twitter: 1ndus
*Not affiliated to any software vendors mentioned in this preso
… and implement few basic things to have a good night’s sleep!
18 slides
Mar, 2009 Barcamp Bangalore 2
What are these?
1. Security
2. Availability & Monitoring
3. Application Errors
4. Backup
5. Source Control
(in order of decreasing priority)
Mar, 2009 Barcamp Bangalore 3
Security Threats
• Your website taken over
• Your database taken over
• Your server taken over
• (Distributed) Denial of Service
Mar, 2009 Barcamp Bangalore 4
Prevention of Security Threats
• Keep your stack up-to-date. Patch.• Establish security-aware coding practice• Know your Logs!• Install open source packages for
preventive/reactive treatments• Get a hardware firewall (if you are popular and
have money)
… Subscribe to Securityfocus alerts
Mar, 2009 Barcamp Bangalore 5
Simple TODO List for You
• Use logwatch and monitor your logs• Make your Database access local (specific IPs
only)• Secure your sshd
– Password-less login, non-default port, no root login
• Use denyhosts to block dictionary SSH attacks (iptables/netfilter is a good bet, I haven’t tried it)
• Close all ports except SSH, HTTP/HTTPS– Use nmap to see what “hackers” see!
Mar, 2009 Barcamp Bangalore 6
A log snapshot of SSH attack Didn't receive an ident from these IPs: 114.200.199.144: 1 Time(s)
Illegal users from: 114.200.199.144: 6 times alias/password: 1 time office/password: 1 time recruit/password: 1 time sales/password: 1 time samba/password: 1 time staff/password: 1 time Failed logins from: 211.60.15.30: 1 time root/password: 1 time 219.137.24.12: 1 time root/password: 1 time
Mar, 2009 Barcamp Bangalore 7
Availability & Monitoring
• Website, Database, SMTP, DNS were down (now up!)
• Poor site performance – Application, Network, or hosting provider?
• CPU, Disk, IO, Memory, Network Interface
• Server down != website down. Put a load balancer
Mar, 2009 Barcamp Bangalore 8
Monitoring – External sample
Mar, 2009 Barcamp Bangalore 9
Monitoring: InternalSystem Level Monitoring with Nagios
Mar, 2009 Barcamp Bangalore 10
Simple TODO List for You
• Do some basic external monitoring – Zoho does url/5minutes at $4/month...cheap!
• Get Nagios for system monitoring
• Use Load Balancer to prevent single server failure– HTTP, Load Balanced database reads
Mar, 2009 Barcamp Bangalore 11
Application Errors
Bad Codefunction validate($key) {
global $weblog ;
if (empty($key)) {
$errorlog->error( "Error : In function validate site key");
return FALSE;
}else{
return TRUE;
}
}
Leads to this in phperror log[13-Feb-2009 09:41:32] PHP Fatal error: Call to a member function error() on a non-object in
/home/padmin/public_html/util/functions.php on line 4
Mar, 2009 Barcamp Bangalore 12
Application Errors
• Simple WARNINGS/FATALs lead to bigger problems– eg. INSERT failed because of duplicate key
(was always inserting 0 for the parameter!)
• Apache error_log may show wrong configuration
• Database logs may show a crash (and auto-recovery!)
Mar, 2009 Barcamp Bangalore 13
Simple TODO for You
• Use a logger like log4j/log4PHP– Modify the handler to send a real-time email
of a desired error level
• Look for Database Error logs, Apache error logs – They will tell you a story!
• Borrow from Security: Use logwatch package
• Review your own application codebase
Mar, 2009 Barcamp Bangalore 14
Backup
• Backup before disaster strikes
• Database backups– Do a dry run of recovery at least once– Ensure consistent, online backups
• Backup your production directories
Mar, 2009 Barcamp Bangalore 15
Simple TODO For You
• (mysql) Use a slave for a consistent backup. No slave? Then Lock the master before dumping
• Take a backup tar of production– Preferably backed up every week, and just
before a deployment and just after a deployment
• Use S3 to store the files remotely
Mar, 2009 Barcamp Bangalore 16
Source Control: Simple TODO For You
• Use SVN– Use hosted… DevGuard..$7/month..cheap!
• Few Developers? Can’t do Linux? No money? Use a local SVN server on Windows. Woorrks! But back-it-up!!!
• Have a prod. deployment strategy– From SVN, DON’T deploy directly on Prod.,
use a separate instance and then scp/rsync over
Mar, 2009 Barcamp Bangalore 17
Summary
• Know Your Logs!
• Be Security aware– Lock your SSH. Close Open Ports
• Do some basic external monitoring
• Backup your Database & prod directory onto a remote location
• Use SVN
Mar, 2009 Barcamp Bangalore 18
Sample Advanced Topics & Thanks!
• Incremental backups, snapshots
• Monitoring Apache Processes, Apache IO, Database connections, Load, Query/sec
• Using SSH Tunneling
• Virtual Private & Public LANs
• VPN
Recommended
