Cliff Click Art of Java Benchmarking

8/12/2019 Cliff Click Art of Java Benchmarking

1/48

A Lock-Free Hash Table

The Art of (Java)Benchmarking

Dr. Cliff ClickChief JVM Architect !isting"ishe# $ngineer blogs%a&"ls'stems%com cliff

A&"l 'stems

J"ne *+, .*.


2/48

///%a&"ls'stems%com

Benchmarking is $as'000

1 An# F"n0001 2M' Java is faster than 'o"r C00031 An# generall' /rong%%%1 Without exception every microbenchmark 45ve seen has

ha# serious fla/s 6 $7ce8t those 45ve ha# a han# in correcting

1 Serious 9 6 2 core3 is "nrelate# to inten#e# meas"rement or 6 error bars e7cee# meas"re# val"es


3/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

8lit o"t micro-bench vs macro-bench

1 Micro-benchmarks are things 'o" /rite 'o"rself 6 Attem8t to #iscover some narro/ targete# fact 6 =enerall' a time# tight loo8 aro"n# some 2/ork3 6 >e8ort score as iterations sec

6 e%g% allocations sec ? ob@ect 8ooling vs =C

1 Macro-benchmarks are s"88ose# to be realistic 6 Larger, longer r"nning

6 e%g% eb erver, !B caching front-en#, ortal A88

6 8ecJBB, 8ecJA88 erver, MLMark, Tra#eD 6 Loa# testing of Eo"r >eal A88


4/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

ome l#er B"ste# Micro-Benchmarks

1 CaffeineMark 2logic3 6 triviall' ma#e #ea# b' J4TG infinite s8ee#"8

1 ciMark Monte-Carlo 6 .I of time in s'nc5# >an#om%ne7t 6 everal foc"se# tests #ea#G infinite s8ee#"8

1 8ecJVM K . K#b ? 8"r8orts to be a !B test 6 >eall' I of time in tring shell-sort

1 8ecJVM K "ntime is m"ch less than * sec


5/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

ome l#er B"ste# Micro-Benchmarks

1 CaffeineMark 2logic3 6 triviall' ma#e #ea# b' J4TG infinite s8ee#"8

1 ciMark Monte-Carlo 6 .I of time in s'nc5# >an#om%ne7t 6 everal foc"se# tests #ea#G infinite s8ee#"8

1 8ecJVM K . K#b ? 8"r8orts to be a !B test 6 >eall' I of time in tring shell-sort

1 8ecJVM K "ntime is m"ch less than * sec


6/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

!ea# Loo8s

// how fast is divide-by-10?long start = Sys.CTM()for( int i=0 i!" i## ) int $ = i/10ret%rn "&1000/(Sys.CTM()-start)

1 Timeline 6 *- 4nter8ret a /hile, ass"me *.ms 6 - J4TG 273 not "se#, loo8 is #ea#, remove#, *.ms 6 N- 24nstantl'3 e7ec"te rest of loo8

1 Time to r"n .ms ? Independent of N! 6 Var' O 99P var' score 99P 2!ial-o- core03


7/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

!ea# Loo8s

1 ometimes J4T 8roves 2res"lts not nee#e#3 6 Then thro/s o"t /hole /ork-loo8 6 After r"nning long eno"gh to J4T 6 o loo8 r"ns at least a little /hile first

1 core 2o8s sec3 not relate# to tri8 co"nt 5O5 6 Larger O 99P larger score

1 core can be infinite- or OaO 6 =enerall' re8orte# as a ver' large, b"t vali# Q

6 An# mi7e# in /ith other n"mbers, conf"sing things 6 (e%g% geomean of infinite5s an# other more real n"mbers)


8/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

ciMark Monte-Carlo

1 .I of time in syn'hroni e 5# >an#om%ne7t1 N-letter-com8an' 2s8amme#3 it b' re8lacing /ith intrinsic

#oing a Com8areAn# /a8 (CA )

1 4 /as or#ere# to follo/ s"it (match 8erformance)1 !o"g Lea sai# 2/ait @"st make a CA from Java31 Hence s"n%misc%AtomicLong /as born

1>a8i#l' re8lace# b' Rnsafe%com8areAn# /a8%%%

1 %%%an# event"all' @ava%lang%AtomicS


9/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice arm"8

1 Co#e starts inter8rete#, then J4T5# 6 J4T5# co#e is *.7 faster than inter8reter

1 J4T5ing ha88ens 2after a /hile3 6 Hot 8ot -server *.,... iterations 6 l"s com8ile time

1 arm"8 co#e /ith some trial r"ns 6 ee8ing testing "ntil r"n-times stabili&e


10/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice arm"8

1 Oot allo/ing /arm"8 is a common mistake1 o8"lar fail"re-mo#e of C-vs-Java com8arisons

6 Fo"n# on man', man', man' /eb 8ages

6 $ntire benchmark r"ns in fe/ milli-secon#s 6 There are #omains reU"iring milli-secon# reboots%%%

1 But most #eskto8 server a88s e78ect 6 >eboots are min"tes long an# #a's a8art 6 tea#'-state thro"gh8"t after /arm"8 is ke' 6 o a benchmark that en#s in *.sec

8robabl' #oes not meas"re an'thing interesting


11/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice 2Com8ile 8lan3

1 J4T makes inlining other com8le7 #ecisions 6 Base# on ver' volatile ran#om #ata 6 4nline #ecisions var' from r"n-to-r"n

1 erformance varies from r"n-to-r"n 6 table n"mbers /ithin a single JVM invocation 6 B"t co"l# var' b' P .I /ith ne/ JVM la"nch 6 Bigger a88s are more 8erformance-stable

1 Micro-benchmarks ten# to be 2fragile3 here 6 e%g% * JVM la"nch in /ill be .I slo/erS

*S tatistically Rigorous Java P erformance Evaluatio n OOPSLA 2007


12/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


8"blic int a() W for(%%%) b()G X8"blic int b() W for(%%%) c()G X8"blic int c() W Y/ork%%% X

A

loo8*Yloo8 Y call C Y @ne loo8 Y @ne loo8* ret"rnC Y ret"rn

80% chance A calls BB inlines C

20% chance A inlines B

B calls C

A

loo8*Y call B @ne loo8* ret"rn

B loo8 %%% @ne loo8 Y ret"rn


13/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 La"nch the JVM man' times 6 Toss * st la"nch to remove caching effects 6 Average o"t 2goo#3 r"ns /ith the 2ba#3 6 !on5t other/ise toss o"tliers

6 ("nless 'o" have goo# reason i%e% "nrelate# loa#)

1 $no"gh times to get statisticall' relevant res"lts 6 Might reU"ire N.Z r"ns

1 >e8ort average and stan#ar# #eviation 6 4n this case, e78ect to see a large st#%#ev

*Statistically rigorous Java performance evaluation OOPSLA 2007


14/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice 2* st fast, n# slo/3

1 Timing harness nee#s to invoke man' targets 6 4n a loo8, re8eate#l' a fe/ times 6 $lse J4T sees * hot target in a loo8

6 An# then #oes a g"ar#e# inline 6 An# then hoists the time# /ork o"tsi#e of timing loo8

'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) stati' final int "=1000000 // illion...stati' int test( ben'h ) + long start = Syste .'%rrentTi eMillis() for( int i=0 i!" i## ) .s,rt(i) // hot loo* v-'all ret%rn "&1000/(Syste .'%rrentTi eMillis()-start)


15/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice 2* st fast, n# slo/3

'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) stati' final int "=1000000 // illion...stati' int test( ben'h ) + long start = Syste .'%rrentTi eMillis() for( int i=0 i!" i## ) .s,rt(i) // hot loo* v-'all ret%rn "&1000/(Syste .'%rrentTi eMillis()-start)

ass in one of t/o#ifferent classes


16/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice v-call o8timi&ation

1 First call test(new ben'h1)

long start = Sys.CTM()for( int i=0 i!" i## ) .s,rt(i)ret%rn "&1000/(Sys.CTM()-start)


17/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 First call test(new ben'h1) 6 ingle target callsiteG J4T #oes g"ar#e# inlining

6 4nlines ben'h1.s,rt


long start = Sys.CTM()for( int i=0 i!" i## )

Math.s,rt(i) // inline ben'h1.s,rtret%rn "&1000/(Sys.CTM()-start)

g"ar#e#inline


18/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 First call test(new ben'h1) 6 ingle target callsiteG J4T #oes g"ar#e# inlining

6 4nlines ben'h1.s,rt 6 Hoists loo8-invariants, #ea#-co#e-remove, etc

6 $7ec"tion time #oes O T #e8en# on O000 6 !rea#e# 2!ial-o- core03


long start = Sys.CTM()for( int i=0 i!" i## )

Math.s,rt(i) // inline ben'h1.s,rtret%rn "&1000/(Sys.CTM()-start)

g"ar#e#inline #ea# co#e

long start = Sys.CTM()

// T2d loo* re oved ret%rn "&1000/(Sys.CTM()-start)


19/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 econ# call test(new ben'h ) 6 n# target of callG g"ar#e# inlining fails 6 Co#e is incorrectG m"st be re-J4T5# 6 Meas"res overhea# of O calls to bench %sUrt

6 l"s g"ar# fail"re, #eo8timi&ation 6 l"s J4T5ing ne/ version of test() 6 l"s virt"al call overhea#

long start = Syste .CTM()for( int i=0 i!" i## ) .s,rt(i)ret%rn "&1000/(Syste .CTM()-start)

int ben'h .s,rt( int i ) + // alternate i *l


20/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 >eversing or#er of calls reverses 2goo#3 2ba#3 6 e%g% 2test(new ben'h ) test(new ben'h1) 3


21/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%



1 Timing harness nee#s to invoke all targets

6 4n a loo8, re8eate#l' a fe/ times

'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) ... // war %* loo* for( int i=0 i!10 i## ) + test( new ben'h1 ) test( new ben'h ) // now try ti ing

*rintf(test(new ben'h1))


22/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%



1 Timing harness nee#s to invoke all targets

6 4n a loo8, re8eate#l' a fe/ times

'lass ben'h1 i *le ents ben'h + void s,rt(int i) 'lass ben'h i *le ents ben'h + void s,rt(int i) ...

// war %* loo* for( int i=0 i!10 i## ) + test( new ben'h1 ) test( new ben'h ) // now try ti ing

*rintf(test(new ben'h1))


23/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice =C

1 Avoi# =C or embrace it1 $ither no (or trivial) allocation, or "se verbose3g' to

make s"re 'o" hit stea#'-state

1 tatistics not @"st average, b"t also st#-#ev1 Look for tren#s

6 Co"l# be cree8ing =C behavior

1 Co"l# be 2leaks3 ca"sing more-/ork-8er-r"n 6 e%g% leak' HashTable gro/ing hea8 or 6 =ro/ing a Linke#List slo/s #o/n searches


24/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice 'nchroni&ation

1 Acco"nt for m"lti-threa#e# locking1 4do see 8eo8le testing, e%g% locking costs on single-

threa#e# 8rograms

1 Never conten#e# lock is ver' chea8 6 ZBiase#Locking makes it even chea8er

1 Very slightly conten#e# lock is 8robabl' +7 more1 Real contention Am#ahl5s La/

6 l"s lots and lots of overhea#1 @ava%"til%conc"rrent is 'o"r frien#


25/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Micro-bench A#vice

1 >ealistic r"ntimes 6 Rnless 'o" nee# s"b-milli-sec reboots

1 arm-"8 loo8s - give the J4T a chance

1 tatistics 8lan for variation in res"lts1 !ea# loo8s ? look for 2!ial-o- core03, #eal /ith it1 * st r"n fast, n# r"n slo/ ? look for it, #eal /ith it1 =C avoi# or embrace


26/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Macro-bench /arnings

1 JVM is too small an'more 6 $as' targetG cache-resi#entG =C ignore#

1 JBB ..., .. 6 Oot m"ch har#er target 6 V$>E 8o8"lar, eas' eno"gh to 2s8am3 6 core rarel' relate# to an'thing real

1 8ecJA88 erver, !aCa8o, 8ecJVM .. , MLMark 6 Bigger, har#er to s8am, less 8o8"lar


27/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

Macro-bench /arnings

1 o8"lar ones are targete# b' com8anies1 =eneral i#ea JVM engineers are honest

6 B"t /ant the best for com8an'

6 o #o targete# o8timi&ations 6 e%g% intrinsic CA for >an#om%ne7t 6 robabl' "sef"l to somebo#' 6 Oever incorrect 6 !efinitel' hel8s this benchmark


28/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

T'8ical erformance T"ning C'cle

1 Benchmark becomes 8o8"lar 1 Management tells $ngineer 24m8rove 5s score031 $ngineer #oes an in-#e8th st"#' of 1 !eci#es o8timi&ation 2E3 /ill hel8

6 An# E is not broken for an'bo#' 6 ossibl' hel8s some other 8rogram

1 4m8lements shi8s a JVM /ith 2E3

1 Management anno"nces score of 2 3 is no/ S1 Rsers 'a/n in #isbelief 2E3 #oes not hel8 them


29/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

8ecJBB ...

1 $mbarrassing 8arallel - no conten#e# locking1 Oo 4 , no #atabase, no ol#-gen =C

6 O T t'8icall' of an' mi##le-/are

6 Very high allocation rate of 'o"ng-gen ob@ects, #efinitel' nott'8icall' 6 B"t ma'be 'o"r 8rogram gets close[

1 e' to 8erformancehaving eno"gh Hea8 to avoi# ol#-gen =C

#"ring +-min time# /in#o/


30/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

8ecJBB ... 8amming

1 !rove T O of s8eciali&e# =C behaviors flags 6 Across man' ven#ors 6 Man' rolle# into 2- ZAggressive 8timi&ations3 6 =oal no ol#-gen =C in + min"tes

1 N-letter-com8an' 2s8amme#3 - /ith a D+-bit VM an# * =ighea8 (in an era of N% = ma7 hea8s)

6 M"ch more allocation, hence 2score3 before =C 6 Oote that /hile h"ge hea8s are genericall' "sef"l to somebo#',

* =ig /as not t'8ical of the time 6 Force# "n to make a D+-bit VM


31/48


32/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

hat can 'o" rea# from the res"lts[

1 The closer 'o"r a88s resemble benchmark 2 3 6 The closer im8rovements to 5s score im8act 'o"

1 H"ge im8rovements to unrelated benchmarks

6 Might be worthless to you1 e%g% 8ecJBB ... is a 8erfect-'o"ng-gen =C test

6 4m8rovements to JBB score have been tie# to better 'o"ng-genbehavior

6 Most /eb-servers s"ffer from L!-gen =C iss"es

6 4m8roving 'o"ng-gen #i#n5t hel8 /eb-servers m"ch


33/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

8ecJBB ..

1 4nten#e# to fi7 JBB ...5s =C iss"es 6 Oo e78licit =C bet/een time# /in#o/s 6 enali&e score if =C 8a"se is too m"ch

( TOs are #ela'e# too long) 6 ame as JBB ..., b"t more ML 6 Oee#s some JavaD-isms o8timi&e#

1 till embarrassing 8arallel ? 'o"ng-gen =C test1 A&"l ran "8 to *


34/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

8ecJBB ..

1 4nten#e# to fi7 JBB ...5s =C iss"es 6 Oo e78licit =C bet/een time# /in#o/s 6 enali&e score if =C 8a"se is too m"ch

( TOs are #ela'e# too long) 6 ame as JBB ..., b"t more ML 6 Oee#s some JavaD-isms o8timi&e#

1 till embarrassing 8arallel ? 'o"ng-gen =C test1 A&"l ran "8 to *


35/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

ome o8"lar Macro-Benchmarks

1 8ecJVM ? too small, no 4 , no =C 6


36/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 8ecJVM ? too small, no 4 , no =C 6


37/48


38/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

8ecJA88 erver

1 Ver' har# to set"8 r"n1 Ver' net/ork, 4 !B intensive1 Oee# a #ecent (not great) JVM (e%g% =C is I)1 B"t 8eak score #e8en#s on an

"ber-!B an# fast #isk or net/ork

1 Oot so heavil' o8timi&e# b' JVM $ngineers1 Lots of 2fle73 in set"8 r"les (!B net/ork config)1 o har# to rea# the res"lts "nless 'o"r e7ternal (non-JVM)

set"8 is similar


39/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

!aCa8o

1 Less 8o8"lar so less o8timi&e#1 >ealistic of mi#-si&e# J a88s1 O T t'8ical of a88-servers, J $$ st"ff 1 $78ect *...5s of classes loa#e# metho#s J4T5#1 ome 4 , more t'8ical =C behavior 1 M"ch better score re8orting r"les

1 !aCa8o "8gra#es coming soon0 6 Oe/ version has /eb-servers 8arallel co#es


40/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

!aCa8o

1 Less 8o8"lar so less o8timi&e#1 >ealistic of mi#-si&e# J a88s1 O T t'8ical of a88-servers, J $$ st"ff 1 $78ect *...5s of classes loa#e# metho#s J4T5#1 ome 4 , more t'8ical =C behavior 1 M"ch better score re8orting r"les

1 !aCa8o "8gra#es coming soon0 6 Oe/ version has /eb-servers 8arallel co#es


41/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 MLMark 6 erf varies b' *.7 base# on ML 8arser J! version 6 Too-/ell-behave# 'o"ng-gen allocation

6 Like !aCa8o ? more realistic of mi#-si&e# J a88s 6 Ver' 8arallel (not a contention benchmark)"nlike most a88-servers

1 8ecJVM .. 6 Also like !aCa8o ? realisticall' si&e# J a88s

6 B"t also has /eb-servers 8arallel a88s 6 Oe/er, not so heavil' targete# b' Ven#ors


42/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%


1 MLMark 6 erf varies b' *.7 base# on ML 8arser J! version 6 Too-/ell-behave# 'o"ng-gen allocation 6

Like !aCa8o ? more realistic of mi#-si&e# J a88s 6 Ver' 8arallel (not a contention benchmark)"nlike most a88-servers

1 8ecJVM .. 6 Also like !aCa8o ? realisticall' si&e# J a88s

6 B"t also has /eb-servers 8arallel a88s 6 Oe/er, not so heavil' targete# b' Ven#ors


43/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

2 o8"lar3 Macro-Benchmark roblems

1 Rnrealistic treatment of =C 6 e%g% Oone in time# /in#o/ 6 r 8erfect 'o"ng-gen collections 6 >eal a88s t'8ical trigger f"ll =C ever' ho"r or so

1 Rnrealistic loa# generation 6 Oot eno"gh loa# to stress s'stem 6 r ver' sim8le or re8etitive loa#s 6 Bottlenecks in getting loa# to server


44/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

2 o8"lar3 Macro-Benchmark roblems

1 Benchmark too short for f"ll =C 6 Man' real a88lications leak

6 Broken N r# 8art' libs, legac' co#e, etc 6 Leaks acc"m"late in ol#-gen

6 hich makes ol#-gen f"ll =C e78ensive 6 B"t benchmark never triggers ol#-gen f"ll =C

1 4 !B not benchmarke# /ell 6 B"t make a h"ge #ifference in >eal Life 6

Eo"r a88 might share 4 !B /ith others


45/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

"mmar'

1 Macrobenchmarks 6 Targete# b' JVM $ngineers

6 Buyer Beware! 6 The closer the benchmark is to 'o"r 8roblem

6 The more likel' im8rovements /ill im8act 'o" 6 =C is likel' to not be typical of real a88lications

6 Your a88lications ever go N% #a's /itho"t a f"ll =C[ 6 4 !B loa# also 8robabl' not t'8ical


46/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

"mmar'

1 Microbenchmarks 6 $as' to rite, Har# to get >ight 6 Easy to be Fooled 6 on5t tell 'o" m"ch abo"t macro-co#e an'/a's 6 ar!u" ? *5s of secon#s to *.5s of secon#s 6 Statistics ? average lots of r"ns

6 $ven o"t variations in the 2com8ile 8lan3 6 Call out to man' metho#s in the hot loo8 6 Be /ar' of dead#code s"8er-score res"lts


47/48

///%a&"ls'stems%com

: ; ..< A&"l 'stems, 4nc%

"t Micro-Tr"st in a Micro-Benchmark0


48/48

///%a&"ls'stems%com

"t Micro-Tr"st in a Micro-Benchmark0

Documents

Cliff Click Art of Java Benchmarking