7/23/2019 JBoss Data Grid Tuning
1/24
JBoss Data Grid 6 Implementations with OpenJDK
John OsborneMiddleware Solutions Architect
Red Hatosborne!redhat"com
#o$ember %&'(
'
mailto:[email protected]:[email protected]7/23/2019 JBoss Data Grid Tuning
2/24
Executive Summary
In recent )ears* I+ departments ha$e been to tas,ed to deal with the per-ormance and scalabilit)
concerns related to the e.ponential /rowth in data" Man) enterprise I+ departments are turnin/ to in0
memor) distributed /rid 1IMDG2 technolo/ies such as JBoss Data Grid in order to meet their Ser$ice
3e$el A/reements 1S3As2" JBoss Data Grid is an enterprise IMDG built o-- o- the communit) open
source communit) proect In-inispan4 it has helped enterprise I+ departments meet their data
per-ormance and scalabilit) re5uirements b) pro$idin/ -eatures li,e e.tremel) -ast 1 O1'2 2 readwrite
access to data* elastic scalin/* hi/h a$ailabilit)* -ault tolerance* and much more" 7hen implementin/ an
IMDG technolo/) li,e JBoss Data Grid per-ormance is essential to e$er) step in the process -rom
de$elopment to production"
+here are se$eral -actors when ta,in/ into consideration the per-ormance o- a distributed cache"
At the hardware le$el it is now commonplace to see $er) lar/e machines capable o- runnin/ J8Ms withlar/e heaps" Ja$a has also e$ol$ed to help to meet the demands o- lar/e heaps throu/h J8M
enhancements includin/ the release o- its new /arba/e collector 1G'2 in Ja$a 9" +he core o- JBoss Data
Grid is a Ja$a application and will re5uire J8M tunin/ whether runnin/ in ser$er mode 1accessin/ the
cache remotel) throu/h protocols li,e R:S+* memcached* Hot Rod2 or in librar) mode 1accessin/ the
cache $ia a local Ja$a A;I2" 7hen usin/ lar/e heaps in Ja$a* the J8M itsel- re5uires a lot o- tunin/ to
-ull) optimi
7/23/2019 JBoss Data Grid Tuning
3/24
Introduction
JBoss Data Grid is an enterprise class* open source* in memor) data /rid plat-orm" It is
implemented as a distributed in0memor) ,e)0$alue #oS@3 store" JBoss Data Grid can be con-i/ured to
run in se$eral di--erence architectures4 the cache supports local* in$alidation* replicated* and distributed
modes" Most -re5uentl)* it is deplo)ed as a data tier to impro$e throu/hput to an application b)
o--loadin/ IO -rom a persistence data store" JBoss Data Grid supports se$eral S@3 and #oS@3 data
persistence options" +he data /rid itsel- acts as a schema0less ,e)$alue store allowin/ the abilit) -or
applications to store distinct obects without ha$in/ a ri/id -i.ed data model"
+he data /rid is t)picall) run on commodit) ser$ers to -orm a cluster that can scale up and
down elasticall) based upon the -luctuatin/ needs o- the application" B) scalin/ elasticall) in replicated
or distributed mode* the /rid is hi/h scalable and -ault tolerant b) ha$in/ no sin/le point o- -ailure" Inthe e$ent o- a ser$er -ailure* JBoss Data Grid pro$ides -ull data inte/rit) throu/h its support -or the
Ja$a +ransaction A;I 1J+A2" It also supports parallel processin/ throu/h its Map Reduce ramewor,
and the abilit) to run tas,s on all or some o- the nodes throu/h its Distributed :.ecution ramewor,"
+here ha$e been a broad ran/e o- sectors that ha$e implemented JBoss Data Grid technolo/) to
power applications* includin/ telecommunications* e0commerce* mobile* /amin/* and -inancial
ser$ices" Re/ardless o- the sector* the main reason to consider a data /rid is per-ormance" 7hen
per-ormance is critical it is necessar) to pa) close attention to the run0time components in order to
ma.imi
7/23/2019 JBoss Data Grid Tuning
4/24
b) pro$idin/ /uidance tips and lessons learned"
Capacity Planning and Sizing The Java Heap
;roper si
7/23/2019 JBoss Data Grid Tuning
5/24
o- a lar/e heap could cause the application to be completel) halted -or a lon/ period o-
time"
Ade5uate heap space is le-t -or JBoss Data Grid and the J8M" JBoss Data Grid will
re5uire additional space -or computations and searches" +he J8M will re5uire additional
space -or critical tas,s li,e /arba/e collection* compaction* temporar) obects* networ,trans-er bu--ers* etc" A /ood rule o- thumb is to ,eep the heap si
7/23/2019 JBoss Data Grid Tuning
6/24
Why Tiered Compilation Is Critical
+he Ja$a compiler has two main tas,s4 enable the code to be plat-orm independent and ensure
hi/h per-ormance" :nablin/ plat-orm independence is accomplish throu/h the use o- a static compiler
li,e a$ac which ta,es Ja$a code and translates it into b)tecode which can be e.ecuted an)where b) the
Ja$a 8irtual Machine" :nsurin/ hi/h per-ormance is not no simple" It is accomplished -irst b)
optimi' compiler which is enabled the F0client J8M -la/" +he >' compiler
is desi/ned -or smaller applications that ha$e -ew resources and are usuall) short li$ed" +he >'
compiler is o-ten bene-icial -or applications that need to -ast start0up times because it d)namicall)
compiles b)tecode $er) earl) on in the applicationLs li-ec)cle* without waitin/ -or e.tensi$e pro-ilin/
and en$ironment data" Since the >' compiler compiles b)te into machine lan/ua/e $er) earl) in the
process* applications usuall) per-orm optimal with the >' compiler soon a-ter the application is
deplo)ed"
Most JBoss Data Grid applications are lon/0runnin/ ser$er0side enterprise applications which
can bene-it -rom e$en -urther compilations" +)picall) -or these t)pes o- applications* it ma,es more
6
7/23/2019 JBoss Data Grid Tuning
7/24
sense to use the >% compiler which waits lon/er to /ather more pro-ilin/ data be-ore optimi' compiler )ields the best results initiall)* the >% compiler t)picall) pro$ides much
better per-ormance in a lon/ runnin/ Ja$a application" +he >% compiler is enabled usin/ the F0ser$er
J8M -la/" +his is the de-ault settin/ -or 6(0bit $ersions o- Ja$a 9" As a matter o- -act* passin/ F0client
to a 6(0bit J8M on 3inu. or 7indows 1not true on Solaris2 will still result in the J8M usin/ the >%
compiler"
One issue with the >% compiler is that it is not tri//ered until the code is e.ecuted '&*&&& times
which is o-ten the reason -or a warm0up period when runnin/ benchmar, tests" +he >' compiler
maintains a lower settin/ o- '*E&&" or both compilers it can be modi-ied with the
0C>ompile+hreshold -la/" +he J8M itsel- uses a counter to ,eep trac, o- when the code should be
compiled" +hat counter decrements o$er time as it mainl) measures how man) times the code has been
e.ecuted recentl)" One option is to simpl) lower the compile threshold but that ris,s -illin/ up the code
cache with machine lan/ua/e that has not been -ull) optimiompilation" +iered compilation combines the best -eatures o- both compilers"
+he >' compiler is most acti$e durin/ application start0up and handles optimi% compiler
deli$ers more ad$anced optimi% compiler alone
9
7/23/2019 JBoss Data Grid Tuning
8/24
and 6MB when usin/ tiered compilation" or Ja$a 9 implementations it is best to increase the code
cache which has a -i.ed si
7/23/2019 JBoss Data Grid Tuning
9/24
collection* stop0the0world pauses can still occur due to -ra/mentation'or when the /arba/e collector
can no lon/er ,eep up with the application obect allocation" Durin/ stop0the0world /arba/e collection
pauses* JBoss Data Grid is completel) halted" I- this occurs durin/ a data trans-er between nodes then
the data bein/ sent across the networ, will start to -ill the networ, bu--ers 1the ne.t section will show
how to modi-) these $alues2" I- the bu--ers -ill up completel)* be-ore the application is resumed* thenpac,et loss will occur which could result in JBoss Data Grid timeouts%" In the worst case* the stabilit)
o- the entire could be a--ected due to especiall) lon/ /arba/e collections" +)picall)* /arba/e collection
per-ormance scales worse than linearl)* meanin/ that when a heap is doubled it would ta,e more than
twice as lon/ -or a -ull /arba/e collection" 7ith $er) lar/e heaps* a -ull /arba/e collection could pause
the application threads -or so lon/ that it would cause the cluster to act as i- the paused node has been
terminated"
+unin/ /arba/e collection -or JBoss Data Grid re5uires care-ul attention to detail and some
anal)sis to see how the application beha$es" ?n-ortunatel)* there is no sil$er bullet to /arba/e
collection as the correct tunin/ will be dependent on application wor,loads" +he /oal o- this section is
to assist de$elopers and architects with the tunin/ and per-ormance considerations o- /arba/e
collection with JBoss Data Grid"
OpenJDK 9 has -our a$ailable /arba/e collectors" +he -irst is the serial /arba/e collector 10C
?seSerialG>2 which uses a sin/le thread to process /arba/e collection and should ne$er be used -or a
hi/h per-ormance Ja$a application" Ma,in/ a decision between the other three /arba/e collectors will
depend on the con-i/uration* re5uirements* and per-ormance testin/"
+he -irst decision to ma,e is whether )ou want to use a throu/hput collector or a concurrent
collector" OpenJDK 9 onl) has one throu/hput collector and it is enabled b) settin/ 0C
?se;arallelG> 0C?se;arallelOldG>" +his is the de-ault collector -or 6(0bit OpenJDK 9 as it
pro$ides the hi/hest throu/hput in an isolated Ja$a application" +he throu/hput collector is e.tremel)
e--icient because it pauses all application threads durin/ minor and -ull /arba/e collections and utili;?s -or ma.imum /arba/e collection throu/hput" 7hen /arba/e collection is completed*
all o- the machine >;?s are used -or application processin/" +he throu/hput collector -ull) compacts
the old /eneration durin/ a -ull G> in order to pre$ent -ra/mentation" JBoss Data Grid applications
'" ra/mentation occurs when the -ree space in the heap is -ra/mented into small pieces o- memor) spread in $arious locations" 7hen-ra/mentation occurs there is o-ten a drastic decrease in per-ormance as the s)stem cannot allocate enou/h conti/uous space -or theapplication" +his causes the /arba/e collector to wor, harder and can o-ten lead to -ull stop0the0world /arba/e collection"
%"JBoss Data Grid timeouts can be modi-ied in the cache con-i/uration"
7/23/2019 JBoss Data Grid Tuning
10/24
usin/ the throu/hput collector will o-ten notice a maor drawbac,4 since it is constantl) stoppin/ all
application threads this will o-ten lead to -re5uent latenc) outliers* especiall) -or lar/er heap si
7/23/2019 JBoss Data Grid Tuning
11/24
application threads durin/ a minor /arba/e collection o- the )oun/ /eneration" >MS collector tries to
a$oid a -ull /arba/e collection b) usin/ bac,/round threads to periodicall) scan throu/h the old
/eneration and discard unused obects" +)picall)* >MS will onl) stop application threads durin/ a
minor /arba/e collection and -or $er) short periods while collectin/ the old /eneration" Howe$er* a
poor tuned >MS collector can still result in e.tremel) lon/ pauses which can hurt the per-ormance o-JBoss Data Grid" In >MS* the bac,/round threads need to scan the entire old /eneration be-ore -reein/
obects" +he time to scan the old /eneration is dependent on heap siMS implementation ma) pro$ide better throu/hput than the G' collector" Howe$er* the worst
case per-ormance with >MS will o-ten be worse than usin/ G' due to ris, o- a concurrent mode
-ailure"
ortunatel)* OpenJDK 9 -ull) supports the G' collector which was desi/ned to do a better ob
handlin/ bac,/round /arba/e collection -or lar/e heaps" Heap -ra/mentation can still occur with the G'
collector* but it is much less li,el) since the G' collector can compact the heap in the bac,/round" +he
G' collector is enabled with the -ollowin/ -la/* 0C?seG'G>" 3i,e* >MS* G' still stops all the
application the threads -or a minor /arba/e collection o- the )oun/ /eneration" G' di$ides the old
/eneration into re/ions and utili
7/23/2019 JBoss Data Grid Tuning
12/24
latenc) 1NCMa.G>;auseMillis2 which de-aults to %&&ms" or lar/e heaps* its best to increase the
latenc) to E&&ms or '&&&ms" +he G' collector should be able to ,eep up with that con-i/uration" One
-inal bene-it to the G' collector o$er the >MS collector is that it re5uires -ar less tunin/" As pre$iousl)
mentioned* JBoss Data Grid does not )et support Ja$a =* but there ha$e been man) impro$ements in
the G' collector -or Ja$a = as well that will be ta,en ad$anta/e o- in -uture release"+he best choice o- a /arba/e collector -or a JBoss Data Grid implementation will $ar) per use
case and can onl) be -ound b) e.perimentin/" Howe$er* most de$elopers and architects o- JBoss Data
Grid are /oin/ to be runnin/ lar/er heaps and there-ore the G' collector should be the collector o-
choice -or hi/h o$erall throu/hput and /ood worst case per-ormance" I- the JBoss Data Grid
implementer is not as concerned with worst case per-ormance and is more concerned with total
throu/hput than a well tuned >MS implementation ma) be a better -it" A /ood wa) to start tunin/ G' is
to set the ma. NCMa.G>;auseMillis to )our pause time -or the worst case S3A" +he throu/hput
collector will be bene-icial in limited use cases and the >MS collector will be bene-icial -or small to
medium si
7/23/2019 JBoss Data Grid Tuning
13/24
increase how man) threads 1and how o-ten the) run2 to ,eep the /arba/e collector one step ahead o- the
application" +he easiest and most e--ecti$e wa) to a$oid concurrent mode -ailures is to ma,e the old
/eneration lar/er" +his can be done b) tunin/ the 0C#ewRatio -la/ which is the ratio o- )oun/
/enerationCold /eneration" +he de-ault settin/ is % which indicates that the old /eneration is % times the
)oun/ /eneration* or % o- the heap siMS collector is o-ten a lar/e balancin/ act" B) settin/ a lar/er )oun/ /eneration siMS b) increasin/ the :den space siMS collector does not collect perm/en" ;erm/en o-ten -ills up when the same
'
7/23/2019 JBoss Data Grid Tuning
14/24
archi$e 1JBoss Data Grid application2 is deplo)ed multiple times to the same runnin/ en$ironment" I-
the perm/en space -ills up a -ull /arba/e collection will be e.ecuted" In order to turn on perm/en
/arba/e collection in >MS use the -ollowin/ -la/sC 0C>MS;ermGenSweepin/:nabled 0C
>MS>lass?nloadin/:nabled"
+he G' collector* i- tuned properl)* should not e.perience a -ull /arba/e collection" G' will stillstop JBoss Data Grid application threads -or minor /arba/e collection and -or some o- the concurrent
/arba/e collection c)cles to compact re/ions in the old /eneration" Similar to >MS* increasin/ the si;auseMillis to a $alue that the G' collector
cannot ,eep up with then it will reduce the )oun/ /eneration si;auseMillis will allow the heap to be si
7/23/2019 JBoss Data Grid Tuning
15/24
Historicall)* OpenJDK has been ,eepin/ class de-initions and class metadata in the ;ermanent
Generation 1;ermGen2 space" +he ;ermGen space is bein/ phased out o- Ja$a 9 and in Ja$a = it has
been completel) replaced with Metaspace" 7hen si
7/23/2019 JBoss Data Grid Tuning
16/24
e$en more important than with an a$era/e siompressin/ the Ja$a obect pointers 10C?se>ompressedOops2 can also help limit the
memor) -ootprint o- obect printers and reduce /arba/e collection" In OpenJDK 9* this settin/ is
alread) enabled when the ma. heap space 1m.2 is set to less than %GB" Since >ompressedOops onl)
compresses re-erences up to %GB* it will not be applicable -or heaps lar/er than %GB"
Additionall)* modi-)in/ soc,et bu--er si
7/23/2019 JBoss Data Grid Tuning
17/24
s)sctl 0w net"core"rmemma.%6%'((&&
s)sctl 0w net"core"wmemma.'&(=E96
JBoss Data Grid utili; connection and handsha,e separatel) -or
each node" JGroups will also handle messa/e deli$er) /uarantee e$en with an unreliable protocol li,e
?D;" Modi-) the JGroups settin/s to match the operatin/ s)stem settin/s" In the JGroups con-i/uration
-ile ensure the -ollowin/ settin/sC
?D;
ucastrec$bu-si;
rec$bu-sis2 to send -rames
that are lar/er than the 'E&& b)te standard and ta,e ad$anta/e o- /reater bandwidth" Man) JBoss Data
Grid implementations will be sendin/ lar/er pac,ets and b) increasin/ the pac,et si
7/23/2019 JBoss Data Grid Tuning
18/24
those runnin/ on '&G networ,s"
:nablin/ Jumbo rames will onl) help JBoss Data Grid per-ormance" Jumbo rames onl)
chan/es the upper limit on the -rame si; and ?D;" ;M+?D mustnot be bloc,ed b) -irewalls or pac,ets could be dropped b) switches that do not support the M+?" On
3inu. )ou can test the si
7/23/2019 JBoss Data Grid Tuning
19/24
onl) be $aluable -or duplicate Strin/s on the same nodes" B) de-ault* JBoss Data Grid stores
obects unseriali
7/23/2019 JBoss Data Grid Tuning
20/24
J8M Health" +he two most important aspects o- J8M Health with respect to JBoss Data Grid is
memor) consumption and /arba/e collection per-ormance" OpenJDK 9 includes se$eral use-ul
monitorin/ tools includin/ console* stat* $isual$m* hat* and map" +hese are critical tools -or
monitorin/ the health o- JBoss Data Grid"
Memor) consumption needs to be closel) monitored" Red Hat recommends that at most E& o-memor) is consumed b) application data so that the rest o- the space can be optimiMS collector which re5uires more manual tunin/" In that case* i- )ou are usin/ the >MS collector*
decrease the 0C#ewRatio settin/ to allow -or a lar/er )oun/ /eneration" I- the application itsel- is
simpl) creatin/ too man) temporar) obects* this ma) also be a /ood time -or a code re$iew"
A /ood wa) to 5uic,l) loo, at the health o- the JBoss Data Grid application is to loo, at a
histo/ram o- the heap distribution" OpenJDK 9 has a couple wa)s to do that -rom the console usin/
%&
7/23/2019 JBoss Data Grid Tuning
21/24
cmd and map" Generatin/ a histo/ram will show the instance count and total si
7/23/2019 JBoss Data Grid Tuning
22/24
console" Jstat also e.poses the collection times usin/ JM so that it can be monitored with other JM
solutions" Jstat wor,s -rom the command line and allows users to see se$eral di--erent $iews o- /arba/e
collection durin/ pro/ram e.ecution" +he command Fstat 0/ccause pid '&&& will continuousl)
print $aluable /arba/e collection in-ormation to the console" I- /arba/e collection needs to be -orced* it
can be done with cmd usin/ the command Fcmd pid G>"run" I- 8isual8M is )our pre-erred toolthen add the 8isualG> plu/in -or $isual /arba/e collection monitorin/"
Additionall)* )ou can collect /arba/e collection lo/s with Ja$a -la/s" +his process has $er) low
o$erhead and can pro$ide a lot o- insi/ht into JBoss Data Grid per-ormance so to enable /arba/e
collection lo//in/ e$en on a production s)stem" or use-ul lo//in/* add the -ollowin/ Ja$a -la/sC F0
$erboseC/c 0lo//cC-ilename 0C;rintG>Details 0C;rintG>+imeStamps* 0C
;rint+enurin/Distribution* 0C;rintG>ApplicationStopped+ime 0C
;rintG>Application>oncurrent+ime " :nsure to update the output -ile in-ormation with the lo//c
-la/" Althou/h )ou can manuall) re$iew the /arba/e collection lo/s* the best wa) to anal) Histo/ram or G>8IW which are separate downloads" I- )ou obser$e a hi/h rate o-
/arba/e collection ma,e sure to re$iew the /arba/e collection tunin/ section"
Althou/h sli/htl) out o- the scope o- this white paper* it is important to also monitor hi/h >;?
utili usin/ open source
tools li,e nicstat which will automaticall) print the utili
7/23/2019 JBoss Data Grid Tuning
23/24
-irst /uideline is to benchmar, usin/ the tar/et con-i/uration" I- JBoss Data Grid will be in production
as a distributed cache* ma,e sure to benchmar, it as a distributed cache as it will beha$e di--erentl)
than as a local cache" +he second /uideline is to ma,e sure )ou ha$e a warm0up period" As noted in the
compilation section* the HotSpot J8M will /enerate machine lan/ua/e -or the -re5uentl) e.ecuted
portions o- the application" As a /eneral rule )ou will want wait at least & seconds and run throu/h atleast '&*&&& iterations o- the code be-ore benchmar,in/ it" +hird* ma,e sure that memor) and /arba/e
collection parameters ha$e been tuned appropriatel)" Additionall)* alwa)s ensure that )our tests are
multi0threaded4 this is especiall) true -or bloc,in/ operations li,e s)nchronous puts into the cache"
Determinin/ the ideal number o- threads will be trial and error but usin/ '&& threads is a /ood startin/
point -or ser$ers with '6 cores" I- the >;? is not near ma.imum capacit) then tr) addin/ more threads"
inall)* it is best to use bul, operations li,e putAll12 as opposed to put12" +his will reduce the number
o- messa/es that need to tra$erse the networ, which will /reatl) increase the throu/hput" Adustin/ the
siloud Ser$in/ Benchmar,in/
1Q>SB2 can also be used to benchmar, JBoss Data Grid in clientser$er mode" Q>SB will test -or
latenc) on the remote JBoss Data Grid ser$er" Apache JMeter can also be used to /enerate load on
JBoss Data Grid in clientser$er mode"
BenchmarC JBoss Data Grid applications using the recommend guidelines' hen possile, use opensource enchmarCing tools to generate load on the data grid'
Conclusion
De$elopin/ and tunin/ -or an in memor) distributed cache re5uires a lot more attention to detail
than a standard Ja$a application" In order to ma.imi
7/23/2019 JBoss Data Grid Tuning
24/24
ma.imum per-ormance usin/ either the >MS or G' collectors" +he >MS collector* i- tuned properl)*
ma) pro$ide the hi/hest throu/hput" Howe$er* the G' collector will pro$ide hi/h throu/hput without
all the tunin/ headaches and will also pro$ide a better worst case scenario than the >MS collector" A
-inel) tuned G' collector will ne$er stop JBoss Data Grid -or a -ull /arba/e collection* and there-ore is
the recommend /arba/e collector o- choice"A JBoss Data Grid application that is producin/ lar/e outliers in response time probabl) needs
tunin/ to the heap si