Upload
90kts
View
12.368
Download
2
Embed Size (px)
Citation preview
VARNISH-CACHE *Lessons Learnt from Performance Testing
* https://www.varnish-cache.orgTuesday, 25 October 11
WHY? FAST!
• If your throughput is currently measured in the hundreds per second, think of varnish in the thousands per second.
• Theoretical limits in hundreds of thousands per second.*
* http://kristianlyng.wordpress.com/2010/10/23/275k-req/Tuesday, 25 October 11
GET OUT OF JAIL. FREE.
• Crappy cache control?
• Expensive CDNs? 20¢/GB?
• Not enough hardware?
• Backend struggling?
Tuesday, 25 October 11
IT WON’T
• Do HTTPS... terminate SSL with something else, hopefully your load balancers.
Tuesday, 25 October 11
INSTALL THE LATEST
• RHEL / CentOS
• Debian
rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-3.0/el5/noarch/varnish-release-3.0-1.noarch.rpmyum install -y varnish
apt-get install varnish
Tuesday, 25 October 11
USE 64BIT*
“It’s all about virtual memory: Things like stack size suddenly matter on 32bit. If you must use Varnish on 32-bit, you’re somewhat on your own. However, try to fit it within 2GB. I wouldn’t recommend a cache larger than 1GB, and no more than a few hundred threads… (Why are you on 32bit again?)”
* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/Tuesday, 25 October 11
SYSTEM CONFIG
• The minimum.
• Do not set “tw_reuse” to 1 (sysctl). It will cause problems for NAT based clients.
• Enable keepalives on load balancers.
echo 'fs.file-max=1000000' >> /etc/sysctl.confecho 'net.ipv4.ip_local_port_range = 1024 65535' >> /etc/sysctl.confsysctl -pecho '* soft nofile 1000000' >> /etc/security/limits.confecho '* hard nofile 1000000' >> /etc/security/limits.confecho 'session required pam_limits.so' >> /etc/pam.d/loginulimit -n 1000000ulimit -n -H
Tuesday, 25 October 11
THREADS
• Adjust default thread* settings in /etc/sysconfig/varnish
thread_pool_add_delay=2thread_pools = <Number of cpu cores>thread_pool_min = <800/number of cpu cores>thread_pool_max = 4000session_linger = 50sess_workspace = <16k to 5m>
* http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/
“The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches. Beyond that, it really doesn’t matter much. Our defaults are good.”
Tuesday, 25 October 11
STORAGE
• Memory is fast, duh.
• Got enough RAM to store the cache?Yes! then use -s mallocUnsure? then use -s file
Tuesday, 25 October 11
VCL
• Varnish Configuration Language
• Looks like C and compiled into C but ...~ “has regex tilda operator”Assignments use set keyword.Only has conditional if tests.Has a bunch of variables and some built in functions.
• Defined by backends, probes, ACLs and subroutines which hook into the varnish workflow.
Tuesday, 25 October 11
VCL
• Default VCL is defined in/etc/varnish/default.vcl
• Then create your custom VCL in -f /var/custom.vclso varnish can fall through to the default.vclCan also include "/var/other.vcl";
• Keep it simple within the VCL workflow*Do most of the work in vcl_recv and vcl_fetch
* https://www.varnish-cache.org/trac/wiki/VCLExampleDefaultTuesday, 25 October 11
LOOKUP AND MISS
• request received from client
• hash request
• lookup object and miss
• fetch response from backend
• deliver to client
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
LOOKUP AND HIT
• request received from client
• hash request
• lookup object and hit
• deliver to client
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
PASS AND DELIVER
• request received from client
• pass through to backend
• fetch response
• deliver to client
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
PIPE
• request received from client
• pipe to backend until client or backend closes
recv
hash
pass
pipe
misshit
fetch
deliver
Tuesday, 25 October 11
PROBES
• Define common health checks for multiple backends.
• window is how many of the latest polls are probed for health.
• threshold is how many polls must succeed to be healthy.
probe health { .url = "/healthCheck?onFailure=true"; .expected_response = 200; .interval = 15s; .timeout = 5s; .window = 4; .threshold = 4; }
Tuesday, 25 October 11
BACKENDS
• Define backend host, port and health checks inline.
• Or named.backend mysite { .host = "www.example.com"; .port = "http"; .probe = "health";}
{ .backend = { .host = "10.10.172.1"; .port = "8001"; .probe = health; } }
Tuesday, 25 October 11
DIRECTORS
• Group backends together using different algorithms. Random Client (based on identity) Hash (based on url) Round robin Fallback
Tuesday, 25 October 11
YMMV
• I found it better to balance in front of Varnish using a hardware load balancer ...
• Random seemed a little too random using Varnish directors however was probably an artefact of the load generation.
Tuesday, 25 October 11
DESIGN DECISIONS
• Share nothing. Easy to scale horizontally.Great if your site is not ‘transactional’.Backend session affinity via LB set cookie.
LB
app
varnishvarnishvarnish
appapp
Set-Cookie=BackendID
Tuesday, 25 October 11
DESIGN DECISIONS
• Share everything. Varnish loves RAM, scale vertically.Varnish cache becomes the shared source of truth.
LB
app
varnish
appapp
Tuesday, 25 October 11
SHARE NOTHING. FALLBACK.
LB
app
varnishvarnishvarnish
appapp
Set-Cookie=VarnishIDdirector varnishA fallback { { .backend = { .host = "a"; } } { .backend = { .host = "b"; } } { .backend = { .host = "c"; } } }
director varnishB fallback { { .backend = { .host = "b"; } } { .backend = { .host = "a"; } } { .backend = { .host = "c"; } } }
director varnishC fallback { { .backend = { .host = "c"; } } { .backend = { .host = "b"; } } { .backend = { .host = "a"; } } }
Tuesday, 25 October 11
TERMINATE SSL.
LB varnish appSet-Cookie=VarnishIDPort=https
:80:443
if (req.http.Port ~ "https") { set req.backend = appa_80;} else { set req.backend = appa_443; }
• Get the LB to set a Port header. Useful when user switches from HTTP to HTTPS so varnish knows which backend port to route to.
Tuesday, 25 October 11
VCL TRICKS
• Rewrite headers to/from production.com.au to testdomain.com
• vcl_recv
• vcl_deliver
set req.http.Host = regsuball(req.http.Host, "testdomain\.com", "com\.au");set req.http.Referer = regsuball(req.http.Host, "testdomain\.com", "com\.au");
set resp.http.Host = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com");set resp.http.Referer = regsuball(resp.http.Host, "\.com\.au", "\.testdomain\.com"); set resp.http.Location = regsuball(resp.http.Location, "\.com\.au", "\.testdomain\.com");set resp.http.Set-Cookie = regsuball(resp.http.Set-Cookie, "Domain=.*;", "");
Tuesday, 25 October 11
VCL TRICKS
• Add some debug headers for hit/miss in firebug in vcl_deliverif (obj.hits > 0) { set resp.http.X-Varnish-Hit = "++"; set resp.http.X-Varnish-Hits = obj.hits; set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");} else { set resp.http.X-Varnish-Hit = "--"; set resp.http.X-Varnish-ServerID = regsub(req.http.Cookie, ".*ServerID=(\d+).*", "\1");}
Tuesday, 25 October 11
VCL TRICKS
• Normalize the Accept-Encoding headers to reduce cache usage in vcl_recv
if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { remove req.http.Accept-Encoding; }}
Tuesday, 25 October 11
VCL TRICKS
• Remove Google Analytics cookies in vcl_recv
• Remember, varnish will pass any request with cookies by default. You will need to over-ride this behaviour if your web app uses cookies.
• Allow lookup of requests with basic auth (in test) or cookies in vcl_recv but beware of creating a per-user-session store.
set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(__[a-z]+|has_js)=[^;]*", "");set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
if (req.http.Authorization || req.http.Cookie) { return (lookup); }
Tuesday, 25 October 11
VCL TRICKS
• Pass on the client IP to the backend servers using the X-Forwarded-For header in vcl_recv
remove req.http.X-Forwarded-For;set req.http.X-Forwarded-For = client.ip;
Tuesday, 25 October 11
VCL TRICKS
• Let varnish keep objects for 2 hours beyond their TTL in vcl_fetch
• Let varnish serve stale objects up to 30 secs old for healthy servers, otherwise 2 hours in vcl_recv
• Can also use saint mode to disable backends for a period of time, otherwise if all backends unavailable, use grace.
set beresp.grace = 2h;
if (! req.backend.healthy) { set req.grace = 2h; } else { set req.grace = 30s; }
Tuesday, 25 October 11
VCL TRICKS
• Force a grace on error. Restart keyword can be called any where and will return request all the way back to vcl_recvsub vcl_error { if (req.restarts == 0) { set req.http.X-Serve-Graced = "1"; return(restart); }}
Tuesday, 25 October 11
TTL
• Varnish will obey cache control headers from backendCache-Control: s-maxageCache-Control: max-age
• Can also override TTL control in VCL, but best do it from the backendset obj.ttl = x
Tuesday, 25 October 11
REDUCING CACHE SIZE
• Normalize / sanitize request headers
• Set Content-Length on backends
• Serve (static) digital assets from VFS cache
• Pipe large objects like videos
Tuesday, 25 October 11
MONITORING VARNISH
• Handy command line toolsvarnishstatvarnishlogvarnishtopvarnishadm
Tuesday, 25 October 11
VARNISHSTAT
• varnishstatHit rate % (x100)Average per secondAverage/sec since bootSum total
• Filteredvarnishstat -f \ client_conn,client_drop,client_req,cache_hit,cache_hitpass,cache_miss,backend_conn,backend_fail
Tuesday, 25 October 11
VARNISHSTAT
• Log all statistics to disk ...
• Then graph as you see fit ...
#!/bin/bashlogdir=/var/log/perf/`hostname`/latestmkdir -p $logdir while truedo varnishstat -x >> "$logdir/varnishstat.xml" sleep 60done
Tuesday, 25 October 11
VARNISHLOG
• varnishlog debugging (-d to dump shared memory) 12 BackendOpen b xx_80[0] x.x.x.x 45546 x.x.x.x 80 12 TxRequest b GET 12 TxURL b / 12 TxProtocol b HTTP/1.1 12 TxHeader b Host: 12 TxHeader b Referer: 12 TxHeader b Cookie: 12 TxHeader b X-Forwarded-For: 1.2.3.4 12 TxHeader b X-Varnish: 2092287889 12 TxHeader b Accept-Encoding: gzip 12 RxProtocol b HTTP/1.1 12 RxStatus b 401 12 RxResponse b Authorization Required 12 RxHeader b Date: Mon, 24 Oct 2011 05:46:18 GMT 12 RxHeader b WWW-Authenticate: Basic realm="XX Authorised Personel Only" 12 RxHeader b Vary: Accept-Encoding 12 RxHeader b Content-Encoding: gzip 12 RxHeader b Content-Length: 336 12 RxHeader b Connection: close 12 RxHeader b Content-Type: text/html; charset=iso-8859-1 12 Fetch_Body b 4 0 1 12 Length b 336 12 BackendClose b xx_80[0]
Tuesday, 25 October 11
VARNISHTOP
• Monitor top URLs hitting backendvarnishtop -i txurl
• Monitor top received URLsvarnishtop -i rxurl
Tuesday, 25 October 11
VARNISHADM
• varnishadm for administration# varnishadmCLI connected to 127.0.0.1 6082200 -----------------------------Varnish Cache CLI 1.0-----------------------------Linux,2.6.32-131.12.1.el6.x86_64,x86_64,-sfile,-smalloc,-hcritbit
Type 'help' for command list.Type 'quit' to close CLI session.
varnish> help200 help [command]ping [timestamp]auth responsequitbannerstatusstartstopvcl.load <configname> <filename>vcl.inline <configname> <quoted_VCLstring>vcl.use <configname>vcl.discard <configname>vcl.listvcl.show <configname>param.show [-l] [<param>]param.set <param> <value>panic.showpanic.clearstorage.listban.url <regexp>ban <field> <operator> <arg> [&& <field> <oper> <arg>]...ban.list
Tuesday, 25 October 11
MORE HELP?
• Talk to the devs, read their blogs.http://kristianlyng.wordpress.comhttp://ingvar.blog.redpill-linpro.com/
• Get some online training.https://www.varnish-software.com/products-services/training/varnish-administration-course
• Consider the Varnish ‘Moral’ License.http://phk.freebsd.dk/VML/
Tuesday, 25 October 11