Designing for garbage collection

Designing for Garbage Collection

Gregg DonovanSenior Software Engineer

Etsy.com

Wednesday, July 31, 13

3.5 Years Search Engineering at Etsy.com

5 years Search & Web Engineeringat TheLadders.com



25+ million members


20+ million items


900k+ active sellers


60+ million monthly unique visitors









CodeAsCraft.etsy.comWednesday, July 31, 13


Understanding GC


Understanding GCMonitoring GC



Debugging Memory Leaks



Debugging Memory LeaksDesign for Partial Availability



public class BuzzwordDetector { static String[] prefixes = { "synergy", "win-win" }; static String[] myArgs = { "clown synergy", "gorilla win-wins", "whamee" };

public static void main(String[] args) { args = myArgs;

int buzzwords = 0; for (int i = 0; i < args.length; i++) { String lc = args[i].toLowerCase(); for (int j = 0; j < prefixes.length; j++) { if (lc.contains(prefixes[j])) { buzzwords++; } } } System.out.println("Found " + buzzwords + " buzzwords"); }}


New(): ref <- allocate() if ref = null /* Heap is full */ collect() ref <- allocate() if ref = null /* Heap is still full */ error "Out of memory" return ref atomic collect(): markFromRoots() sweep(HeapStart, HeapEnd)

From Garbage Collection HandbookWednesday, July 31, 13

markFromRoots(): initialise(worklist) for each fld in Roots ref <- *fld if ref != null && not isMarked(ref) setMarked(ref) add(worklist, ref) mark() initialise(worklist): worklist <- empty mark(): while not isEmpty(worklist) ref <- remove(worklist) /* ref is marked */ for each fld in Pointers(ref) child <- *fld if (child != null && not isMarked(child) setMarked(child) add(worklist, child)

From Garbage Collection HandbookWednesday, July 31, 13

Trivia: Who invented the first GC and Mark-and-Sweep?


Weak Generational Hypothesis


Where do objects in your application live?


GC Terminology:Concurrent vs Parallel


JVM Collectors


Serial


Throughput


CMS


Garbage First (G1)


Continuously Concurrent Compacting Collector (C4)


IBM, Dalvik, etc.?


Why Throughput?


Questions so far?


Monitoring


GC time per request


...import java.lang.management.*;...

public static long getCollectionTime() { long collectionTime = 0; for (GarbageCollectorMXBean mbean : ManagementFactory.getGarbageCollectorMXBeans()) { collectionTime += mbean.getCollectionTime(); } return collectionTime; }

Available via JMX



Visual GC




export GC_DEBUG="-verbose:gc \-XX:+PrintGCDateStamps \-XX:+PrintHeapAtGC \-XX:+PrintGCApplicationStoppedTime \-XX:+PrintGCApplicationConcurrentTime \-XX:+PrintAdaptiveSizePolicy \-XX:AdaptiveSizePolicyOutputInterval=1 \-XX:+PrintTenuringDistribution \-XX:+PrintGCDetails \-XX:+PrintCommandLineFlags \-XX:+PrintSafepointStatistics \-Xloggc:/var/log/search/gc.log"



2013-04-08T20:14:00.162+0000: 4197.791: [Full GCAdaptiveSizeStart: 4206.559 collection: 213 PSAdaptiveSizePolicy::compute_generation_free_space limits: desired_promo_size: 9927789154 promo_limit: 8321564672 free_in_old_gen: 4096 max_old_gen_size: 22190686208 avg_old_live: 22190682112AdaptiveSizePolicy::compute_generation_free_space limits: desired_eden_size: 9712028790 old_eden_size: 8321564672 eden_limit: 8321564672 cur_eden: 8321564672 max_eden_size: 8321564672 avg_young_live: 7340911616AdaptiveSizePolicy::compute_generation_free_space: gc time limit gc_cost: 1.000000 GCTimeLimit: 98PSAdaptiveSizePolicy::compute_generation_free_space: costs minor_time: 0.167092 major_cost: 0.965075 mutator_cost: 0.000000 throughput_goal: 0.990000 live_space: 29859940352 free_space: 16643129344 old_promo_size: 8321564672 old_eden_size: 8321564672 desired_promo_size: 8321564672 desired_eden_size: 8321564672AdaptiveSizeStop: collection: 213 [PSYoungGen: 8126528K->7599356K(9480896K)] [ParOldGen: 21670588K->21670588K(21670592K)] 29797116K->29269944K(31151488K) [PSPermGen: 58516K->58512K(65536K)], 8.7690670 secs] [Times: user=137.36 sys=0.03, real=8.77 secs] Heap after GC invocations=213 (full 210): PSYoungGen total 9480896K, used 7599356K [0x00007fee47ab0000, 0x00007ff0dd000000, 0x00007ff0dd000000) eden space 8126528K, 93% used [0x00007fee47ab0000,0x00007ff0177ef080,0x00007ff037ac0000) from space 1354368K, 0% used [0x00007ff037ac0000,0x00007ff037ac0000,0x00007ff08a560000) to space 1354368K, 0% used [0x00007ff08a560000,0x00007ff08a560000,0x00007ff0dd000000) ParOldGen total 21670592K, used 21670588K [0x00007fe91d000000, 0x00007fee47ab0000, 0x00007fee47ab0000) object space 21670592K, 99% used [0x00007fe91d000000,0x00007fee47aaf0e0,0x00007fee47ab0000) PSPermGen total 65536K, used 58512K [0x00007fe915000000, 0x00007fe919000000, 0x00007fe91d000000) object space 65536K, 89% used [0x00007fe915000000,0x00007fe918924130,0x00007fe919000000)}


GC Log Analyzers?

GCHisto

GCViewer

garbagecat

github.com/Netflix/gcvizWednesday, July 31, 13

Graphing with Logster

github.com/etsy/logster


http://github.com/etsy/logster



GC Dashboardgithub.com/etsy/dashboard





YourKit.com


Designing for Partial Availability


JVMTI GC Hook?


How can a client ignore GC-ing hosts?


Server lies to clients about availability

TCP socket receive buffer

TCP write buffer


“Banner” protocol1. Connect via TCP

2. Wait ~1-10ms

3. Either receive magic four byte header or try another host

4. Only send query after receiving header from server


0xC0DEA5CF


public function open() { $this->handle_ = @fsockopen($this->host_, $this->port_, $errno, $errstr, $this->connectTimeout_ / 1000.0); try { stream_set_timeout($this->handle_, 0, $banner_timeout * 1000); $read_start = microtime(true); $data = $this->readAll(4); $read_time = (microtime(true) - $read_start) * 1000; // micros to millis $arr = unpack('N', $data); $value = $arr[1]; if ($value !== 0xC0DEA5CF) { StatsD::increment("search.baddata.{$short_hostname}.{$this->getPort()}"); throw new TTransportException("[$value] does match banner [0xC0DEA5CF]"); } } catch (Exception $e) { $this->close(); // this won't necessarily be closed by clients throw new TTransportException($message, self::BANNER_TIMEOUT_CODE); }}


private static class BannerSendingTProcessorFactory extends TProcessorFactory { private final TProcessor base; public BannerSendingTProcessorFactory(TProcessor base) { super(base); this.base = base; }

@Override public TProcessor getProcessor(TTransport trans) { return new BannerTProcessor(base, (TSocket) trans); }}

private static final class BannerTProcessor implements TProcessor { private final TProcessor base; private final TSocket tsocket;

private BannerTProcessor(TProcessor base, TSocket tsocket) { this.base = checkNotNull(base); this.tsocket = checkNotNull(tsocket); }

@Override public boolean process(TProtocol in, TProtocol out) throws TException { this.tsocket.write(TBannerUtil.BANNER, 0, 4); this.tsocket.flush(); return this.base.process(in, out); }}


What if GC happens mid-request?


Backup requests


Jeff Dean: Achieving Rapid Response Time in Large

Online Services


Sharding?

Naive approach: only as fast as the slowest shard.


“Make a reliable whole out of unreliable parts.”


Memory Leaks


SolrIndexSearcher generation marking with YourKit triggers



Questions so far?


Miscellaneous Topics


System.gc()?


-XX:+UseCompressedOops


-XX:+UseNUMA


Paging


#!/usr/bin/env bash

# This script is designed to be run every minute by cron.

host=$(hostname -s)

psout=$(ps h -p `cat /var/run/etsy-search.pid` -o min_flt,maj_flt 2>/dev/null)min_flt=$(echo $psout | awk '{print $1}') # minor page faultsmaj_flt=$(echo $psout | awk '{print $2}') # major page faults

epoch_s=$(date +%s)

echo -e "search_memstats.$host.etsy-search.min_flt\t${min_flt:-0}\t$epoch_s" | nc graphite.etsycorp.com 2003echo -e "search_memstats.$host.etsy-search.maj_flt\t${maj_flt:-0}\t$epoch_s" | nc graphite.etsycorp.com 2003


Solution 1: Buy more RAM

Ideally enough RAM to:Keep data in OS file buffersAND ensure no paging of VM memory AND whatever else happens on the box

~$5-10/GB


echo “0” > /proc/sys/vm/swappiness


mlock()/mlockall()

github.com/LucidWorks/mlockall-agent


echo “-17” > /proc/$PID/oom_adj

Mercy from the OOM Killer


Huge Pages


-XX:+AlwaysPreTouch


Future Directions


Many small VMs instead of one large VM

microsharding


Off-heap memory with sun.misc.Unsafe?


Try G1 again


Try C4 again


Resources


gchandbook.org



bit.ly/mmgcb

Mark Miller’s GC Bootcamp


http://bit.ly/mmgcb

http://bit.ly/mmgcb

bit.ly/giltene

Gil Tene: Understanding Java Garbage Collection


http://bit.ly/mmgcb

http://bit.ly/mmgcb

bit.ly/cpumemory

Ulrich Drepper: What Every Programmer Should Know About Memory


http://bit.ly/mmgcb

http://bit.ly/mmgcb

github.com/pingtimeout/jvm-optionsWednesday, July 31, 13

Read the JVM Source(Not as scary as it sounds.)

hg.openjdk.java.net/jdk7/jdk7


Mechanical Sympathy Google Group

bit.ly/mechsym


Questions?

Thanks for coming!

Gregg [email protected]


mailto:[email protected]

mailto:[email protected]

Technology

Designing for garbage collection