How a major ISP built a new anti-abuse platformMike O’Reirdan
Comcast Distinguished EngineerInternet Systems EngineeringComcast National Engineering & Technical Operations
Outline
Comcast facts and figures
Why build a new platform
Fundamentals of anti spam
Size of the problem
Previous approach
Current solution
Migration methods
Current status
Why a new platform?
Moved from a hosted to an in-house platformNeed to improve customer experience by further reducing volumes of spam to the mailboxDeploy a platform which can economically and easily scaleEmerging threats in abuse landscape
• Image spam• Botnets• VoIP spam (SPIT)
Need to have a plug-and-play architecture• Firmly believe that no one vendor will be the best forever• We need a mix of vendors and approaches to hedge our bets and
reduce risk• Somebody in this room may be our next vendor when you have
gone from the lab to the VC and into beta
Size of the problem
Volumes of spam are astronomical• 596 million connection attempts (Jan25th 2008)
• 539 million connection attempts rejected• 93% spam• 76 million messages delivered
Connection attempts increases massively above this around holidays such as Thanksgiving.The problems is criminality at massive scale
Fundamentals of anti-spam
Not much differentiation between major mail box hosters and other ISPs with regard to spam percentages and volumesThree stages
• Blocking based on IP (reputation and DUL space)– 5% of CPU cycles– Removes ~70% of the spam
• Blocking based on message protocol and heuristics– 10% of CPU cycles– Removes ~15% of the spam
• Blocking based on content– 85% of CPU cycles– Remove ~10% of the spam
Idea is to use the least cycles to remove the most messages
Previous approach
100s of Linux blade servers• No site fail over
Multiple RBLs using BIND for DNS
Heuristics and protocol filtering
Spam content filtering using industry standard software
Virus filtering using industry standard software
New Approach
Fewer Linux Blade servers distributed over two sites• Full dual site redundancy with each site fully capable of
carrying 100% of trafficRBLs hosted on a specialised DNS based platform
• Trend• Spamhaus• Return Path
Protocol and heuristics filtering performed on the Bizanga IMP MTAs which run on LinuxSpam content filtering technologyAnti-virus technology
Heuristics employed
Directory Harvest attack
Dictionary attack
rDNS check
Throttling
Dynamic space blocking
Non-existent user block
Content filtering-detecting spammy content
Cloudmark• Relies on multiple sources of data
– Spam / no Spam reports from end users– Honeypots
• Initially based on Vipul’s Razor• Applies algorithmically derived signatures to
incoming email (Proprietary)• Zero hour anti virus
Trend Anti-virus• Signature analysis• Heuristics
Migration
Relatively simple process to migrate from old platformMoved traffic across by re-pointing comcast.net MX records to new platform and making lots of involved highly planned DNS configuration changesPerformed a series of increasing short duration burst test scaleThen moved 5% of the traffic. After platform rules proved stable, traffic was moved across in slightly larger increments over several days to the new platform.This method allowed us to quickly revert back (under 30 minutes) to old platform in the event of any issues without customer impact
Lessons learned
It always helps to be able to test the new platform against an existing live e-mail flow but this is difficult at our scale with a multi-Gbps mail flowFailing that, heavy reliance has to be placed on cooperation with vendors and existing platform technology usersRules used on an old platform do not always map across neatly to a new one