Rick ReedWhatsApp

Half a Billion 600M+Unsuspecting FreeBSD Users

topics

historystatisticsarchitecturewinschallenges

rr

Joined WhatsApp server team in 2011Performance, multimedia, security, opsPreviously: Yahoo!, SGI, Metaphor, AionFirst FreeBSD: 2.2.8 at Yahoo!

Why FreeBSD?

Previous Yahoo! experienceHigh-quality, high-performance TCP stackScaleStability

Numbers

Over 600M monthly users140M concurrent connections440k connections per second1.1M messages sent per second1B pics sent per day

Numbers

~800 servers (incl. 200 chat + 350 mms)

Performance

SMP scaling is greatkernel relatively free of contentionErlang (actor model)BEAM (Erlang VM) very SMP-aware

System metrics

Hardware (typical)

SoftLayer bare-metal1U/2U/4U Supermicrodual E5-2690v2 (40 threads)(public, private) x 2 lagg x 1g igb (some 10g ix)64-768GB RAM1-12 800GB SSD (12x4TB HDD for video)JBOD w/ mobo, Adaptec, and LSI controllers

Software Platform

FreeBSD 9.1 - 9.3ufsgmirror for HDD video serversstarting to evaluate 10.1

Software Platform

Erlang R16B01 (w/ our patchset)rtpriopatches address specific contention issuesschedulers bound to cpusallocators tuned for superpage promotionyaws for web-serving (but disable sendfile)

Software Platform

stud for SSL termination (proxy via loopback)

Data storage

Erlang mnesiaaccount data, group membership, etc.in-memory + dump/log persistence

Flat-file on hashed directory treeoffline mailboxes, mediafs layout tuned per workload

System metrics

6 x 800GB SSD per offline storage server

Versions

FreeBSD 7: 2009-2010goodbugs: some

Versions

FreeBSD 8: 2011-2012betterbugs: zero-copy sockets, igb, tsc tctuning got us to ~2.8M conns/server

Versions

FreeBSD 9: 2013-yes!bugs: very, very rare panicenhancements:

>2G dirhash sizemax tcp rexmit interval

Tuning

kernconfremove unused modulesenable debugging/tracing

Tuning

loader varsigb (numq=8, rx_process_limit=-1, r/txd=4k)kern.nbuf=128kkern.max{files,filesperproc,proc}net.inet.tcp.tcbhashsize=512knet.inet.tcp.hostcache.bucketlimit=1

Tuning

Networkcapacity (mbufs, sockets, etc.)

Tuning

vfsufs.dirhash_maxmem (patched for >2G)dirty buffer limits & sync delays

Issues

ufs panics (fs inconsistencies after crash/pfail)bursty syncer behaviorfiledesc table contentionpcb table contention (1.5m igb + 3m loopback)

Now vs. then

machdep.panic_on_nmiVM pressure

Wishlist

Multipath TCPFaster large RAM boot

Questions?