8/12/2019 Cliff Click Art of Java Benchmarking
1/48
A Lock-Free Hash Table
The Art of (Java)Benchmarking
Dr. Cliff ClickChief JVM Architect !isting"ishe# $ngineer blogs%a&"ls'stems%com cliff
A&"l 'stems
J"ne *+, .*.
8/12/2019 Cliff Click Art of Java Benchmarking
2/48
///%a&"ls'stems%com
Benchmarking is $as'000
1 An# F"n0001 2M' Java is faster than 'o"r C00031 An# generall' /rong%%%1 Without exception every microbenchmark 45ve seen has
ha# serious fla/s 6 $7ce8t those 45ve ha# a han# in correcting
1 Serious 9 6 2 core3 is "nrelate# to inten#e# meas"rement or 6 error bars e7cee# meas"re# val"es
8/12/2019 Cliff Click Art of Java Benchmarking
3/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
8lit o"t micro-bench vs macro-bench
1 Micro-benchmarks are things 'o" /rite 'o"rself 6 Attem8t to #iscover some narro/ targete# fact 6 =enerall' a time# tight loo8 aro"n# some 2/ork3 6 >e8ort score as iterations sec
6 e%g% allocations sec ? ob@ect 8ooling vs =C
1 Macro-benchmarks are s"88ose# to be realistic 6 Larger, longer r"nning
6 e%g% eb erver, !B caching front-en#, ortal A88
6 8ecJBB, 8ecJA88 erver, MLMark, Tra#eD 6 Loa# testing of Eo"r >eal A88
8/12/2019 Cliff Click Art of Java Benchmarking
4/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ome l#er B"ste# Micro-Benchmarks
1 CaffeineMark 2logic3 6 triviall' ma#e #ea# b' J4TG infinite s8ee#"8
1 ciMark Monte-Carlo 6 .I of time in s'nc5# >an#om%ne7t 6 everal foc"se# tests #ea#G infinite s8ee#"8
1 8ecJVM K . K#b ? 8"r8orts to be a !B test 6 >eall' I of time in tring shell-sort
1 8ecJVM K "ntime is m"ch less than * sec
8/12/2019 Cliff Click Art of Java Benchmarking
5/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ome l#er B"ste# Micro-Benchmarks
1 CaffeineMark 2logic3 6 triviall' ma#e #ea# b' J4TG infinite s8ee#"8
1 ciMark Monte-Carlo 6 .I of time in s'nc5# >an#om%ne7t 6 everal foc"se# tests #ea#G infinite s8ee#"8
1 8ecJVM K . K#b ? 8"r8orts to be a !B test 6 >eall' I of time in tring shell-sort
1 8ecJVM K "ntime is m"ch less than * sec
8/12/2019 Cliff Click Art of Java Benchmarking
6/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
!ea# Loo8s
// how fast is divide-by-10?long start = Sys.CTM()for( int i=0 i!" i## ) int $ = i/10ret%rn "&1000/(Sys.CTM()-start)
1 Timeline 6 *- 4nter8ret a /hile, ass"me *.ms 6 - J4TG 273 not "se#, loo8 is #ea#, remove#, *.ms 6 N- 24nstantl'3 e7ec"te rest of loo8
1 Time to r"n .ms ? Independent of N! 6 Var' O 99P var' score 99P 2!ial-o- core03
8/12/2019 Cliff Click Art of Java Benchmarking
7/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
!ea# Loo8s
1 ometimes J4T 8roves 2res"lts not nee#e#3 6 Then thro/s o"t /hole /ork-loo8 6 After r"nning long eno"gh to J4T 6 o loo8 r"ns at least a little /hile first
1 core 2o8s sec3 not relate# to tri8 co"nt 5O5 6 Larger O 99P larger score
1 core can be infinite- or OaO 6 =enerall' re8orte# as a ver' large, b"t vali# Q
6 An# mi7e# in /ith other n"mbers, conf"sing things 6 (e%g% geomean of infinite5s an# other more real n"mbers)
8/12/2019 Cliff Click Art of Java Benchmarking
8/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ciMark Monte-Carlo
1 .I of time in syn'hroni e 5# >an#om%ne7t1 N-letter-com8an' 2s8amme#3 it b' re8lacing /ith intrinsic
#oing a Com8areAn# /a8 (CA )
1 4 /as or#ere# to follo/ s"it (match 8erformance)1 !o"g Lea sai# 2/ait @"st make a CA from Java31 Hence s"n%misc%AtomicLong /as born
1>a8i#l' re8lace# b' Rnsafe%com8areAn# /a8%%%
1 %%%an# event"all' @ava%lang%AtomicS
8/12/2019 Cliff Click Art of Java Benchmarking
9/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice arm"8
1 Co#e starts inter8rete#, then J4T5# 6 J4T5# co#e is *.7 faster than inter8reter
1 J4T5ing ha88ens 2after a /hile3 6 Hot 8ot -server *.,... iterations 6 l"s com8ile time
1 arm"8 co#e /ith some trial r"ns 6 ee8ing testing "ntil r"n-times stabili&e
8/12/2019 Cliff Click Art of Java Benchmarking
10/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice arm"8
1 Oot allo/ing /arm"8 is a common mistake1 o8"lar fail"re-mo#e of C-vs-Java com8arisons
6 Fo"n# on man', man', man' /eb 8ages
6 $ntire benchmark r"ns in fe/ milli-secon#s 6 There are #omains reU"iring milli-secon# reboots%%%
1 But most #eskto8 server a88s e78ect 6 >eboots are min"tes long an# #a's a8art 6 tea#'-state thro"gh8"t after /arm"8 is ke' 6 o a benchmark that en#s in *.sec
8robabl' #oes not meas"re an'thing interesting
8/12/2019 Cliff Click Art of Java Benchmarking
11/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice 2Com8ile 8lan3
1 J4T makes inlining other com8le7 #ecisions 6 Base# on ver' volatile ran#om #ata 6 4nline #ecisions var' from r"n-to-r"n
1 erformance varies from r"n-to-r"n 6 table n"mbers /ithin a single JVM invocation 6 B"t co"l# var' b' P .I /ith ne/ JVM la"nch 6 Bigger a88s are more 8erformance-stable
1 Micro-benchmarks ten# to be 2fragile3 here 6 e%g% * JVM la"nch in /ill be .I slo/erS
*S tatistically Rigorous Java P erformance Evaluatio n OOPSLA 2007
8/12/2019 Cliff Click Art of Java Benchmarking
12/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice 2Com8ile 8lan3
8"blic int a() W for(%%%) b()G X8"blic int b() W for(%%%) c()G X8"blic int c() W Y/ork%%% X
A
loo8*Yloo8 Y call C Y @ne loo8 Y @ne loo8* ret"rnC Y ret"rn
80% chance A calls BB inlines C
20% chance A inlines B
B calls C
A
loo8*Y call B @ne loo8* ret"rn
B loo8 %%% @ne loo8 Y ret"rn
8/12/2019 Cliff Click Art of Java Benchmarking
13/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice 2Com8ile 8lan3
1 La"nch the JVM man' times 6 Toss * st la"nch to remove caching effects 6 Average o"t 2goo#3 r"ns /ith the 2ba#3 6 !on5t other/ise toss o"tliers
6 ("nless 'o" have goo# reason i%e% "nrelate# loa#)
1 $no"gh times to get statisticall' relevant res"lts 6 Might reU"ire N.Z r"ns
1 >e8ort average and stan#ar# #eviation 6 4n this case, e78ect to see a large st#%#ev
*Statistically rigorous Java performance evaluation OOPSLA 2007
8/12/2019 Cliff Click Art of Java Benchmarking
14/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice 2* st fast, n# slo/3
1 Timing harness nee#s to invoke man' targets 6 4n a loo8, re8eate#l' a fe/ times 6 $lse J4T sees * hot target in a loo8
6 An# then #oes a g"ar#e# inline 6 An# then hoists the time# /ork o"tsi#e of timing loo8
'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) stati' final int "=1000000 // illion...stati' int test( ben'h ) + long start = Syste .'%rrentTi eMillis() for( int i=0 i!" i## ) .s,rt(i) // hot loo* v-'all ret%rn "&1000/(Syste .'%rrentTi eMillis()-start)
8/12/2019 Cliff Click Art of Java Benchmarking
15/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice 2* st fast, n# slo/3
'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) stati' final int "=1000000 // illion...stati' int test( ben'h ) + long start = Syste .'%rrentTi eMillis() for( int i=0 i!" i## ) .s,rt(i) // hot loo* v-'all ret%rn "&1000/(Syste .'%rrentTi eMillis()-start)
ass in one of t/o#ifferent classes
8/12/2019 Cliff Click Art of Java Benchmarking
16/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 First call test(new ben'h1)
long start = Sys.CTM()for( int i=0 i!" i## ) .s,rt(i)ret%rn "&1000/(Sys.CTM()-start)
8/12/2019 Cliff Click Art of Java Benchmarking
17/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 First call test(new ben'h1) 6 ingle target callsiteG J4T #oes g"ar#e# inlining
6 4nlines ben'h1.s,rt
long start = Sys.CTM()for( int i=0 i!" i## ) .s,rt(i)ret%rn "&1000/(Sys.CTM()-start)
long start = Sys.CTM()for( int i=0 i!" i## )
Math.s,rt(i) // inline ben'h1.s,rtret%rn "&1000/(Sys.CTM()-start)
g"ar#e#inline
8/12/2019 Cliff Click Art of Java Benchmarking
18/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 First call test(new ben'h1) 6 ingle target callsiteG J4T #oes g"ar#e# inlining
6 4nlines ben'h1.s,rt 6 Hoists loo8-invariants, #ea#-co#e-remove, etc
6 $7ec"tion time #oes O T #e8en# on O000 6 !rea#e# 2!ial-o- core03
long start = Sys.CTM()for( int i=0 i!" i## ) .s,rt(i)ret%rn "&1000/(Sys.CTM()-start)
long start = Sys.CTM()for( int i=0 i!" i## )
Math.s,rt(i) // inline ben'h1.s,rtret%rn "&1000/(Sys.CTM()-start)
g"ar#e#inline #ea# co#e
long start = Sys.CTM()
// T2d loo* re oved ret%rn "&1000/(Sys.CTM()-start)
8/12/2019 Cliff Click Art of Java Benchmarking
19/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 econ# call test(new ben'h ) 6 n# target of callG g"ar#e# inlining fails 6 Co#e is incorrectG m"st be re-J4T5# 6 Meas"res overhea# of O calls to bench %sUrt
6 l"s g"ar# fail"re, #eo8timi&ation 6 l"s J4T5ing ne/ version of test() 6 l"s virt"al call overhea#
long start = Syste .CTM()for( int i=0 i!" i## ) .s,rt(i)ret%rn "&1000/(Syste .CTM()-start)
int ben'h .s,rt( int i ) + // alternate i *l
8/12/2019 Cliff Click Art of Java Benchmarking
20/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 >eversing or#er of calls reverses 2goo#3 2ba#3 6 e%g% 2test(new ben'h ) test(new ben'h1) 3
8/12/2019 Cliff Click Art of Java Benchmarking
21/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 >eversing or#er of calls reverses 2goo#3 2ba#3 6 e%g% 2test(new ben'h ) test(new ben'h1) 3
1 Timing harness nee#s to invoke all targets
6 4n a loo8, re8eate#l' a fe/ times
'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) ... // war %* loo* for( int i=0 i!10 i## ) + test( new ben'h1 ) test( new ben'h ) // now try ti ing
*rintf(test(new ben'h1))
8/12/2019 Cliff Click Art of Java Benchmarking
22/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice v-call o8timi&ation
1 >eversing or#er of calls reverses 2goo#3 2ba#3 6 e%g% 2test(new ben'h ) test(new ben'h1) 3
1 Timing harness nee#s to invoke all targets
6 4n a loo8, re8eate#l' a fe/ times
'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) ...
// war %* loo* for( int i=0 i!10 i## ) + test( new ben'h1 ) test( new ben'h ) // now try ti ing
*rintf(test(new ben'h1))
8/12/2019 Cliff Click Art of Java Benchmarking
23/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice =C
1 Avoi# =C or embrace it1 $ither no (or trivial) allocation, or "se verbose3g' to
make s"re 'o" hit stea#'-state
1 tatistics not @"st average, b"t also st#-#ev1 Look for tren#s
6 Co"l# be cree8ing =C behavior
1 Co"l# be 2leaks3 ca"sing more-/ork-8er-r"n 6 e%g% leak' HashTable gro/ing hea8 or 6 =ro/ing a Linke#List slo/s #o/n searches
8/12/2019 Cliff Click Art of Java Benchmarking
24/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice 'nchroni&ation
1 Acco"nt for m"lti-threa#e# locking1 4do see 8eo8le testing, e%g% locking costs on single-
threa#e# 8rograms
1 Never conten#e# lock is ver' chea8 6 ZBiase#Locking makes it even chea8er
1 Very slightly conten#e# lock is 8robabl' +7 more1 Real contention Am#ahl5s La/
6 l"s lots and lots of overhea#1 @ava%"til%conc"rrent is 'o"r frien#
8/12/2019 Cliff Click Art of Java Benchmarking
25/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Micro-bench A#vice
1 >ealistic r"ntimes 6 Rnless 'o" nee# s"b-milli-sec reboots
1 arm-"8 loo8s - give the J4T a chance
1 tatistics 8lan for variation in res"lts1 !ea# loo8s ? look for 2!ial-o- core03, #eal /ith it1 * st r"n fast, n# r"n slo/ ? look for it, #eal /ith it1 =C avoi# or embrace
8/12/2019 Cliff Click Art of Java Benchmarking
26/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Macro-bench /arnings
1 JVM is too small an'more 6 $as' targetG cache-resi#entG =C ignore#
1 JBB ..., .. 6 Oot m"ch har#er target 6 V$>E 8o8"lar, eas' eno"gh to 2s8am3 6 core rarel' relate# to an'thing real
1 8ecJA88 erver, !aCa8o, 8ecJVM .. , MLMark 6 Bigger, har#er to s8am, less 8o8"lar
8/12/2019 Cliff Click Art of Java Benchmarking
27/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
Macro-bench /arnings
1 o8"lar ones are targete# b' com8anies1 =eneral i#ea JVM engineers are honest
6 B"t /ant the best for com8an'
6 o #o targete# o8timi&ations 6 e%g% intrinsic CA for >an#om%ne7t 6 robabl' "sef"l to somebo#' 6 Oever incorrect 6 !efinitel' hel8s this benchmark
8/12/2019 Cliff Click Art of Java Benchmarking
28/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
T'8ical erformance T"ning C'cle
1 Benchmark becomes 8o8"lar 1 Management tells $ngineer 24m8rove 5s score031 $ngineer #oes an in-#e8th st"#' of 1 !eci#es o8timi&ation 2E3 /ill hel8
6 An# E is not broken for an'bo#' 6 ossibl' hel8s some other 8rogram
1 4m8lements shi8s a JVM /ith 2E3
1 Management anno"nces score of 2 3 is no/ S1 Rsers 'a/n in #isbelief 2E3 #oes not hel8 them
8/12/2019 Cliff Click Art of Java Benchmarking
29/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
8ecJBB ...
1 $mbarrassing 8arallel - no conten#e# locking1 Oo 4 , no #atabase, no ol#-gen =C
6 O T t'8icall' of an' mi##le-/are
6 Very high allocation rate of 'o"ng-gen ob@ects, #efinitel' nott'8icall' 6 B"t ma'be 'o"r 8rogram gets close[
1 e' to 8erformancehaving eno"gh Hea8 to avoi# ol#-gen =C
#"ring +-min time# /in#o/
8/12/2019 Cliff Click Art of Java Benchmarking
30/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
8ecJBB ... 8amming
1 !rove T O of s8eciali&e# =C behaviors flags 6 Across man' ven#ors 6 Man' rolle# into 2- ZAggressive 8timi&ations3 6 =oal no ol#-gen =C in + min"tes
1 N-letter-com8an' 2s8amme#3 - /ith a D+-bit VM an# * =ighea8 (in an era of N% = ma7 hea8s)
6 M"ch more allocation, hence 2score3 before =C 6 Oote that /hile h"ge hea8s are genericall' "sef"l to somebo#',
* =ig /as not t'8ical of the time 6 Force# "n to make a D+-bit VM
8/12/2019 Cliff Click Art of Java Benchmarking
31/48
8/12/2019 Cliff Click Art of Java Benchmarking
32/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
hat can 'o" rea# from the res"lts[
1 The closer 'o"r a88s resemble benchmark 2 3 6 The closer im8rovements to 5s score im8act 'o"
1 H"ge im8rovements to unrelated benchmarks
6 Might be worthless to you1 e%g% 8ecJBB ... is a 8erfect-'o"ng-gen =C test
6 4m8rovements to JBB score have been tie# to better 'o"ng-genbehavior
6 Most /eb-servers s"ffer from L!-gen =C iss"es
6 4m8roving 'o"ng-gen #i#n5t hel8 /eb-servers m"ch
8/12/2019 Cliff Click Art of Java Benchmarking
33/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
8ecJBB ..
1 4nten#e# to fi7 JBB ...5s =C iss"es 6 Oo e78licit =C bet/een time# /in#o/s 6 enali&e score if =C 8a"se is too m"ch
( TOs are #ela'e# too long) 6 ame as JBB ..., b"t more ML 6 Oee#s some JavaD-isms o8timi&e#
1 till embarrassing 8arallel ? 'o"ng-gen =C test1 A&"l ran "8 to *
8/12/2019 Cliff Click Art of Java Benchmarking
34/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
8ecJBB ..
1 4nten#e# to fi7 JBB ...5s =C iss"es 6 Oo e78licit =C bet/een time# /in#o/s 6 enali&e score if =C 8a"se is too m"ch
( TOs are #ela'e# too long) 6 ame as JBB ..., b"t more ML 6 Oee#s some JavaD-isms o8timi&e#
1 till embarrassing 8arallel ? 'o"ng-gen =C test1 A&"l ran "8 to *
8/12/2019 Cliff Click Art of Java Benchmarking
35/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ome o8"lar Macro-Benchmarks
1 8ecJVM ? too small, no 4 , no =C 6
8/12/2019 Cliff Click Art of Java Benchmarking
36/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ome o8"lar Macro-Benchmarks
1 8ecJVM ? too small, no 4 , no =C 6
8/12/2019 Cliff Click Art of Java Benchmarking
37/48
8/12/2019 Cliff Click Art of Java Benchmarking
38/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
8ecJA88 erver
1 Ver' har# to set"8 r"n1 Ver' net/ork, 4 !B intensive1 Oee# a #ecent (not great) JVM (e%g% =C is I)1 B"t 8eak score #e8en#s on an
"ber-!B an# fast #isk or net/ork
1 Oot so heavil' o8timi&e# b' JVM $ngineers1 Lots of 2fle73 in set"8 r"les (!B net/ork config)1 o har# to rea# the res"lts "nless 'o"r e7ternal (non-JVM)
set"8 is similar
8/12/2019 Cliff Click Art of Java Benchmarking
39/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
!aCa8o
1 Less 8o8"lar so less o8timi&e#1 >ealistic of mi#-si&e# J a88s1 O T t'8ical of a88-servers, J $$ st"ff 1 $78ect *...5s of classes loa#e# metho#s J4T5#1 ome 4 , more t'8ical =C behavior 1 M"ch better score re8orting r"les
1 !aCa8o "8gra#es coming soon0 6 Oe/ version has /eb-servers 8arallel co#es
8/12/2019 Cliff Click Art of Java Benchmarking
40/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
!aCa8o
1 Less 8o8"lar so less o8timi&e#1 >ealistic of mi#-si&e# J a88s1 O T t'8ical of a88-servers, J $$ st"ff 1 $78ect *...5s of classes loa#e# metho#s J4T5#1 ome 4 , more t'8ical =C behavior 1 M"ch better score re8orting r"les
1 !aCa8o "8gra#es coming soon0 6 Oe/ version has /eb-servers 8arallel co#es
8/12/2019 Cliff Click Art of Java Benchmarking
41/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ome o8"lar Macro-Benchmarks
1 MLMark 6 erf varies b' *.7 base# on ML 8arser J! version 6 Too-/ell-behave# 'o"ng-gen allocation
6 Like !aCa8o ? more realistic of mi#-si&e# J a88s 6 Ver' 8arallel (not a contention benchmark)"nlike most a88-servers
1 8ecJVM .. 6 Also like !aCa8o ? realisticall' si&e# J a88s
6 B"t also has /eb-servers 8arallel a88s 6 Oe/er, not so heavil' targete# b' Ven#ors
8/12/2019 Cliff Click Art of Java Benchmarking
42/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
ome o8"lar Macro-Benchmarks
1 MLMark 6 erf varies b' *.7 base# on ML 8arser J! version 6 Too-/ell-behave# 'o"ng-gen allocation 6
Like !aCa8o ? more realistic of mi#-si&e# J a88s 6 Ver' 8arallel (not a contention benchmark)"nlike most a88-servers
1 8ecJVM .. 6 Also like !aCa8o ? realisticall' si&e# J a88s
6 B"t also has /eb-servers 8arallel a88s 6 Oe/er, not so heavil' targete# b' Ven#ors
8/12/2019 Cliff Click Art of Java Benchmarking
43/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
2 o8"lar3 Macro-Benchmark roblems
1 Rnrealistic treatment of =C 6 e%g% Oone in time# /in#o/ 6 r 8erfect 'o"ng-gen collections 6 >eal a88s t'8ical trigger f"ll =C ever' ho"r or so
1 Rnrealistic loa# generation 6 Oot eno"gh loa# to stress s'stem 6 r ver' sim8le or re8etitive loa#s 6 Bottlenecks in getting loa# to server
8/12/2019 Cliff Click Art of Java Benchmarking
44/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
2 o8"lar3 Macro-Benchmark roblems
1 Benchmark too short for f"ll =C 6 Man' real a88lications leak
6 Broken N r# 8art' libs, legac' co#e, etc 6 Leaks acc"m"late in ol#-gen
6 hich makes ol#-gen f"ll =C e78ensive 6 B"t benchmark never triggers ol#-gen f"ll =C
1 4 !B not benchmarke# /ell 6 B"t make a h"ge #ifference in >eal Life 6
Eo"r a88 might share 4 !B /ith others
8/12/2019 Cliff Click Art of Java Benchmarking
45/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
"mmar'
1 Macrobenchmarks 6 Targete# b' JVM $ngineers
6 Buyer Beware! 6 The closer the benchmark is to 'o"r 8roblem
6 The more likel' im8rovements /ill im8act 'o" 6 =C is likel' to not be typical of real a88lications
6 Your a88lications ever go N% #a's /itho"t a f"ll =C[ 6 4 !B loa# also 8robabl' not t'8ical
8/12/2019 Cliff Click Art of Java Benchmarking
46/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
"mmar'
1 Microbenchmarks 6 $as' to rite, Har# to get >ight 6 Easy to be Fooled 6 on5t tell 'o" m"ch abo"t macro-co#e an'/a's 6 ar!u" ? *5s of secon#s to *.5s of secon#s 6 Statistics ? average lots of r"ns
6 $ven o"t variations in the 2com8ile 8lan3 6 Call out to man' metho#s in the hot loo8 6 Be /ar' of dead#code s"8er-score res"lts
8/12/2019 Cliff Click Art of Java Benchmarking
47/48
///%a&"ls'stems%com
: ; ..< A&"l 'stems, 4nc%
"t Micro-Tr"st in a Micro-Benchmark0
8/12/2019 Cliff Click Art of Java Benchmarking
48/48
///%a&"ls'stems%com
"t Micro-Tr"st in a Micro-Benchmark0