Upload
mgarg
View
223
Download
0
Embed Size (px)
Citation preview
8/10/2019 201403 Tampa Performance3
1/99
RED HAT CONFIDENTIAL | JEREMY EDER1
Performance Tuning and
Anal!i! of Red Ha"
En"er#ri!e Linu$ % and &
Jeremy Eder
Principal Software EngineerMarch 13, 2014
8/10/2019 201403 Tampa Performance3
2/99
RED HAT CONFIDENTIAL | JEREMY EDER2
Agenda' Performance Anal!i! of RHEL%(&
Performance Engineering O)er)ie* Performance Anal!i! +"ili"ie!
Tuned
N+MA Tuning
Po*er Managemen"
8/10/2019 201403 Tampa Performance3
3/99
RED HAT CONFIDENTIAL | JEREMY EDER3
Performance EngineeringO)er)ie*
Micro,-enc.mar/!A##lica"ion!(-enc.mar/!
8/10/2019 201403 Tampa Performance3
4/99RED HAT CONFIDENTIAL | JEREMY EDER4
Red Ha" Confiden"ial
Red Ha" Performance Engineering
Benchmark ! code path co"erage #P$ ! linpack, lm%ench
Memory ! lm%ench, Mc#alpin Stream
&ik '( ! 'o)one, aiotre ! ci, *#, iS#S'
*ileytem ! '()one, potmark! e+t34, +f- gf2,gl.ter
/etwork ! /etperf ! 10 %it, 40 %it 'B, P#'3
Bare Metal, E5 67M
8hite %o+ 9M&'ntel, with o.r (EM partner
8/10/2019 201403 Tampa Performance3
5/99RED HAT CONFIDENTIAL | JEREMY EDER:
Performance Pro0ec"! ( Tooling
E-: ;n.mad< ;t.na
8/10/2019 201403 Tampa Performance3
6/99
RED HAT CONFIDENTIAL | JEREMY EDER
Performance O#"imi1a"ion
Ou",of,".e,2o$
8/10/2019 201403 Tampa Performance3
7/99
RED HAT CONFIDENTIAL | JEREMY EDER
Performance Tuning Au"oma"ion
9.tomatic =.ning
=.ned
=ranparent .gepage
n.mad
irF%alance
Man.al =.ning
/9
Static .gepage
/$M9 Pinning n.mactlC
'> Pinning t.naC
E5
n.maG%alancing
8/10/2019 201403 Tampa Performance3
8/99
RED HAT CONFIDENTIAL | JEREMY EDERH
-u"333*.a" if *e .a)e a #ro2lem 4
9.tomatic not eno.gh---
8/10/2019 201403 Tampa Performance3
9/99
RED HAT CONFIDENTIAL | JEREMY EDER10
-u"333*.a" if *e .a)e a #ro2lem 4
9.tomatic not eno.gh--- /eed to eek o.t the lat I percent
8/10/2019 201403 Tampa Performance3
10/99
RED HAT CONFIDENTIAL | JEREMY EDER11
-u"333*.a" if *e .a)e a #ro2lem 4
9.tomatic not eno.gh--- /eed to eek o.t the lat I percent
/eed &eterminim
8/10/2019 201403 Tampa Performance3
11/99
RED HAT CONFIDENTIAL | JEREMY EDER12
-u"333*.a" if *e .a)e a #ro2lem 4
9.tomatic not eno.gh--- /eed to eek o.t the lat I percent
/eed &eterminim
(6---here? where it get? f.n
8/10/2019 201403 Tampa Performance3
12/99
RED HAT CONFIDENTIAL | JEREMY EDER13
O)er)ie* of PerformanceAnal!i! +"ili"ie!
#erf
f
8/10/2019 201403 Tampa Performance3
13/99
RED HAT CONFIDENTIAL | JEREMY EDER14
#erf
$erpace tool to read #P$
co.nter and kernel tracepoint
f li
8/10/2019 201403 Tampa Performance3
14/99
RED HAT CONFIDENTIAL | JEREMY EDER1:
#erf li!"
it co.ntertracepoint a"aila%le
on yo.r ytem
f li "
8/10/2019 201403 Tampa Performance3
15/99
RED HAT CONFIDENTIAL | JEREMY EDER1
#erf li!"
grep for omething intereting,
may%e to ee what n.ma%alance idoing K
---more on n.ma%alance later---
f "
8/10/2019 201403 Tampa Performance3
16/99
RED HAT CONFIDENTIAL | JEREMY EDER15
#erf "o#
Sytem@wide ?top? "iew of %.y
f.nction
8/10/2019 201403 Tampa Performance3
17/99
#erf record
8/10/2019 201403 Tampa Performance3
18/99
RED HAT CONFIDENTIAL | JEREMY EDER1H
#erf record
ecord ytem@wide @aC
9 ingle command
#erf record
8/10/2019 201403 Tampa Performance3
19/99
RED HAT CONFIDENTIAL | JEREMY EDER20
#erf record
ecord ytem@wide @aC
9 ingle command9n e+iting proce @pC
#erf record
8/10/2019 201403 Tampa Performance3
20/99
RED HAT CONFIDENTIAL | JEREMY EDER21
#erf record
ecord ytem@wide @aC
9 ingle command9n e+iting proce @pC
9dd call@chain recording @gC
#erf record
8/10/2019 201403 Tampa Performance3
21/99
RED HAT CONFIDENTIAL | JEREMY EDER22
#erf record
ecord ytem@wide @aC
9 ingle command9n e+iting proce @pC
9dd call@chain recording @gC(nly pecific e"ent @eC
#erf record
8/10/2019 201403 Tampa Performance3
22/99
RED HAT CONFIDENTIAL | JEREMY EDER23
#erf record
ecord ytem@wide @aC
9 ingle command9n e+iting proce @pC
9dd call@chain recording @gC(nly pecific e"ent @eC
#erf re#or"
8/10/2019 201403 Tampa Performance3
23/99
RED HAT CONFIDENTIAL | JEREMY EDER24
#erf re#or"
de")ero
#erf re#or"
8/10/2019 201403 Tampa Performance3
24/99
RED HAT CONFIDENTIAL | JEREMY EDER2:
#erf re#or"
de")ero
oflagLdirect
#erf diff
8/10/2019 201403 Tampa Performance3
25/99
RED HAT CONFIDENTIAL | JEREMY EDER2
#erf diff
#ompare 2 perf recording
f 2 5d i i 6
8/10/2019 201403 Tampa Performance3
26/99
RED HAT CONFIDENTIAL | JEREMY EDER25
#erf #ro2e 5dnamic "race#oin"!6
'nert a tracepoint on any f.nction---
=ry ?perf pro%e @*? to lit poi%ilitie
Cool7 *.a" el!e a go" 4
8/10/2019 201403 Tampa Performance3
27/99
RED HAT CONFIDENTIAL | JEREMY EDER2
Cool7 *.a" el!e a go" 4
8/10/2019 201403 Tampa Performance3
28/99
RED HAT CONFIDENTIAL | JEREMY EDER2H
O)er)ie* of PerformanceAnal!i! +"ili"ie!
Performance Co,Pilo" 5PCP6
8/10/2019 201403 Tampa Performance3
29/99
#mc.ar" 8 gra#.ical me"ric #lo""ing "ool
8/10/2019 201403 Tampa Performance3
30/99
RED HAT CONFIDENTIAL | JEREMY EDER31
#mc.ar" gra#.ical me"ric #lo""ing "ool
#an plot myriad performance tatitic
#mc.ar" 8 gra#.ical me"ric #lo""ing "ool
8/10/2019 201403 Tampa Performance3
31/99
RED HAT CONFIDENTIAL | JEREMY EDER32
#mc.ar" gra#.ical me"ric #lo""ing "ool
#an plot myriad performance tatitic
ecording mode allow for replay
i-e- on a different ytem
ecord in $', then
# pmafm $recording.folio
#mc.ar" 8 gra#.ical me"ric #lo""ing "ool
8/10/2019 201403 Tampa Performance3
32/99
RED HAT CONFIDENTIAL | JEREMY EDER33
#mc.ar" gra#.ical me"ric #lo""ing "ool
#an plot myriad performance tatitic
ecording mode allow for replay
i-e- on a different ytem
ecord in $', then
# pmafm $recording.folio
Ship with many pre@cooked ;"iew
8/10/2019 201403 Tampa Performance3
33/99
RED HAT CONFIDENTIAL | JEREMY EDER34
9.a" do ".e!e )ie*! loo/ li/e 4
Performance Co,Pilo" Demo :cri#"
8/10/2019 201403 Tampa Performance3
34/99
RED HAT CONFIDENTIAL | JEREMY EDER3:
Performance Co Pilo" Demo :cri#"
# CPU
/root/pig -s 5
# DISKdd if=/dev/zero of=/root/2GB count=2!" s=$% of&'g=direct
# ()*+,K
netperf -. &' -& 5
# %)%,0
/root/pig -1 $3"! -& s&eep -s 5
=iny cript to e+ercie 4 food gro.p---
8/10/2019 201403 Tampa Performance3
35/99
RED HAT CONFIDENTIAL | JEREMY EDER3
#P$
oad 9"g
'(PS
/etwork
Memory9llocated
collec"l mode
8/10/2019 201403 Tampa Performance3
36/99
RED HAT CONFIDENTIAL | JEREMY EDER35
#P$
8/10/2019 201403 Tampa Performance3
37/99
collec"l mode
8/10/2019 201403 Tampa Performance3
38/99
RED HAT CONFIDENTIAL | JEREMY EDER3H
'(PS
/E=
#P$
collec"l mode
8/10/2019 201403 Tampa Performance3
39/99
RED HAT CONFIDENTIAL | JEREMY EDER40
'(PS
MEM
/E=
#P$
a"o# mode
8/10/2019 201403 Tampa Performance3
40/99
RED HAT CONFIDENTIAL | JEREMY EDER41
a"o# mode
;ue!"ion! !o far 4
8/10/2019 201403 Tampa Performance3
41/99
RED HAT CONFIDENTIAL | JEREMY EDER42
;ue!"ion! !o far 4
8/10/2019 201403 Tampa Performance3
42/99
RED HAT CONFIDENTIAL | JEREMY EDER43
N+MA Tuning
Di!co)er
8/10/2019 201403 Tampa Performance3
43/99
RED HAT CONFIDENTIAL | JEREMY EDER44
# g #
ow can ' "i.ali)e my ytem? /$M9topology in ed at Enterprie in.+K
httpDacce-redhat-comiteol.tion25H
/$M9 /ode 0
8/10/2019 201403 Tampa Performance3
44/99
RED HAT CONFIDENTIAL | JEREMY EDER4:
ow can ' "i.ali)e my ytem? /$M9topology in ed at Enterprie in.+K
httpDacce-redhat-comiteol.tion25H
/$M9 /ode 0 /$M9 /ode 1
8/10/2019 201403 Tampa Performance3
45/99
RED HAT CONFIDENTIAL | JEREMY EDER4
ow can ' "i.ali)e my ytem? /$M9topology in ed at Enterprie in.+K
httpDacce-redhat-comiteol.tion25H
P#' &e"ice
/$M9 /ode 0 /$M9 /ode 1
8/10/2019 201403 Tampa Performance3
46/99
RED HAT CONFIDENTIAL | JEREMY EDER45
# lscpu
Architecture: x86_64...CPUs!: "6nline CPUs! list: %"&
'hreads! per core: "Cores! per soc(et: 8)oc(ets!: *+U,A nodes!: *...
+U,A node% CPUs!: %-+U,A node" CPUs!: 8"&
ogical #ore=
8/10/2019 201403 Tampa Performance3
47/99
RED HAT CONFIDENTIAL | JEREMY EDER4
# lscpu
Architecture: x86_64...CPUs!: "6nline CPUs! list: %"&
'hreads! per core: "Cores! per soc(et: 8)oc(ets!: *+U,A nodes!: *...
+U,A node% CPUs!: %-+U,A node" CPUs!: 8"&
ogical #ore=
/$M9 /ode#ore Map
N+MA To#olog and PCI -u!
8/10/2019 201403 Tampa Performance3
48/99
RED HAT CONFIDENTIAL | JEREMY EDER4H
'ntall adapter ;cloe< to the #P$ that will r.n theperformance critical application-
8hen B'(S report locality, irF%alance handle/$M9'> affinity a.tomatically-
# lstoponographics egrep /+U,Aeth4/ +U,A+ode 0#% P#% "4412!
+U,A+ode 0#" P#" "4412!
+et 0#"% 3eth43
RHEL N+MA :c.eduler
8/10/2019 201403 Tampa Performance3
49/99
RED HAT CONFIDENTIAL | JEREMY EDER:0
E
n.mactl, n.matat enhancement
n.mad ! .ermode tool, dynamically monitor, [email protected]
E5 ! n.ma%alance Ena%le &ia%le
yctl kernel-n.ma%alancingLN0,1O
Ho* "o manage N+MA manuall , C.ec/li!"
8/10/2019 201403 Tampa Performance3
50/99
RED HAT CONFIDENTIAL | JEREMY EDER:1
#hecklit
eearch =opology
Make a reo.rce plan
#onider '(
7irt.ali)ation
=ool
ltopolcp.
cgro.p, n.mactl
irF%alanceP#' B.
n.mat.nen.mad
Ho* "o manage N+MA manuall , C.ec/li!"
8/10/2019 201403 Tampa Performance3
51/99
RED HAT CONFIDENTIAL | JEREMY EDER:2
#hecklit
eearch =opology
Make a reo.rce plan
#onider '(
7irt.ali)ation
=ool
ltopolcp.
cgro.p, n.mactl
irF%alanceP#' B.
n.mat.nen.mad
Ho* "o manage N+MA manuall , C.ec/li!"
8/10/2019 201403 Tampa Performance3
52/99
RED HAT CONFIDENTIAL | JEREMY EDER:3
#hecklit
eearch =opology
Make a reo.rce plan
#onider '(
7irt.ali)ation
=ool
ltopolcp.
cgro.p, n.mactl
irF%alanceP#' B.
n.mat.nen.mad
8/10/2019 201403 Tampa Performance3
53/99
Ho* "o manage N+MA manuall , C.ec/li!"
8/10/2019 201403 Tampa Performance3
54/99
RED HAT CONFIDENTIAL | JEREMY EDER::
#hecklit
eearch =opology
Make a reo.rce plan
#onider '(
7irt.ali)ation
=ool
ltopolcp.
cgro.p, n.mactl
irF%alanceP#' B.
n.mat.nen.mad
N+MA Tool!' numa!"a"
8/10/2019 201403 Tampa Performance3
55/99
RED HAT CONFIDENTIAL | JEREMY EDER:
#ompletely rewritten for E-4
Per@node procmeminfoBackward compati%le
N+MA Tool!' numa!"a"
8/10/2019 201403 Tampa Performance3
56/99
RED HAT CONFIDENTIAL | JEREMY EDER:5
#ompletely rewritten for E-4
Per@node procmeminfoBackward compati%le
numa!"a"' com#a"i2ili" mode 5old6
8/10/2019 201403 Tampa Performance3
57/99
RED HAT CONFIDENTIAL | JEREMY EDER:
# numastat
node% node"
numa_hit --&8--5 ""55%%4*numa_miss % %numa_foreign % %
interleae_hit %*&4 %%55local_node 65%*-"% "*5&""6%other_node 8*8&%*5 *4-868*
numa!"a"' #er,node meminfo 5ne*6# numastat -mczs
d 0
8/10/2019 201403 Tampa Performance3
58/99
RED HAT CONFIDENTIAL | JEREMY EDER:H
Node 0------
MemTotal 65491MemFree 60366MemUsed 5124Actie 2650File!a"es 2021
Actie$ile% 16&6Actieanon% 964Anon!a"es 964'nactie 341'nactie$ile% 340
(la) 3&0(*eclaima)le 20&(Unreclaim 1+3Anon,u"e!a"es 134
numa!"a"' #er,node meminfo 5ne*6# numastat -mczs
N d 0 N d 1 T t l
8/10/2019 201403 Tampa Performance3
59/99
RED HAT CONFIDENTIAL | JEREMY EDER0
Node 0 Node 1 Total ------ ------ ------
MemTotal 65491 65536 13102+MemFree 60366 59+33 120099MemUsed 5124 5&03 1092+Actie 2650 2&2+ 54++File!a"es 2021 3216 523&
Actie$ile% 16&6 22++ 3963Actieanon% 964 551 1515Anon!a"es 964 550 1514'nactie 341 946 12&+'nactie$ile% 340 946 12&6
(la) 3&0 43& &1&(*eclaima)le 20& 20+ 415(Unreclaim 1+3 230 403Anon,u"e!a"es 134 236 3+0
8/10/2019 201403 Tampa Performance3
60/99
RED HAT CONFIDENTIAL | JEREMY EDER1
N+MA Tuning
numad
N+MA' Proce!! :c.eduler -e.a)ior
S h d l di t i% t l d l ll
8/10/2019 201403 Tampa Performance3
61/99
RED HAT CONFIDENTIAL | JEREMY EDER2
Sched.ler ditri%.te load e"enly acro all core
Maintain reponi"ene
(ptimi)ing for #P$ .tili)ation
=rie to .e idle #P$, regardle of where procememory i located
B$=
$ing remote memory degrade
performance
Ho* "o manage N+MA manuall , C.ec/li!"
8/10/2019 201403 Tampa Performance3
62/99
RED HAT CONFIDENTIAL | JEREMY EDER3
Short =erm E-4C
$erpace ol.tion
n.mad
ong =erm E5C
ed at ha taken 2 approache for c.tomerD
Ho* "o manage N+MA manuall , C.ec/li!"
8/10/2019 201403 Tampa Performance3
63/99
RED HAT CONFIDENTIAL | JEREMY EDER4
Short =erm E-4C
$erpace ol.tion
n.mad
ong =erm E5C
'n@6ernel Sol.tion
n.ma%alance
ed at ha taken 2 approache for c.tomerD
9.a" doe! i" loo/ li/e 4
8/10/2019 201403 Tampa Performance3
64/99
RED HAT CONFIDENTIAL | JEREMY EDER:
Effec" of Au"oma"ic N+MA -alancing :of"*are
8/10/2019 201403 Tampa Performance3
65/99
RED HAT CONFIDENTIAL | JEREMY EDER
# numastat c pig default scheduler 7 nonoptimal!
P9 +ode % +ode "*&-8 pig! *"* ""8-8*&-5 pig! "588 "*%"*&8% pig! "4%%% "*&8" pig! "58" "*%*%
Before n.mad
Effec" of Au"oma"ic N+MA -alancing :of"*are
8/10/2019 201403 Tampa Performance3
66/99
RED HAT CONFIDENTIAL | JEREMY EDER5
# numastat c pig default scheduler 7 nonoptimal!
P9 +ode % +ode "*&-8 pig! *"* ""8-8*&-5 pig! "588 "*%"*&8% pig! "4%%% "*&8" pig! "58" "*%*%
P9 +ode % +ode "*&-8 pig! "4%%% %*&-5 pig! % "4%%%*&8% pig! "4%%% %
*&8" pig! % "4%%%
Before n.mad
9fter n.mad
Effec" of numad(numa2alance
9.tomatic /$M9 Balancing @ /$M9&
8/10/2019 201403 Tampa Performance3
67/99
RED HAT CONFIDENTIAL | JEREMY EDER
1 2 3 4 : 5 H 10 11 12 13 14 1: 1
0
2000
4000
000
000
10000
12000
14000
/(&E@0@MB /(&E@1@MB
=ime econdC
Mega%ytePer/$M9
/ode
n.mad %egin
n.mad done
;ue!"ion! on N+MA 4
8/10/2019 201403 Tampa Performance3
68/99
RED HAT CONFIDENTIAL | JEREMY EDERH
8/10/2019 201403 Tampa Performance3
69/99
RED HAT CONFIDENTIAL | JEREMY EDER50
"uned
9.a" i! ="uned> 4
8/10/2019 201403 Tampa Performance3
70/99
RED HAT CONFIDENTIAL | JEREMY EDER51
=.ning profile deli"ery mechanim
ed at hip tuned profilesthatimpro"e performance for manyworkload---hopef.lly yo.r
Ye!7 2u" *. do I care 4
8/10/2019 201403 Tampa Performance3
71/99
RED HAT CONFIDENTIAL | JEREMY EDER52
Tuned' :"orage Performance -oo!"
8/10/2019 201403 Tampa Performance3
72/99
RED HAT CONFIDENTIAL | JEREMY EDER53
arger i %etter
Tuned' Ne"*or/ La"enc Performance -oo!"
8/10/2019 201403 Tampa Performance3
73/99
RED HAT CONFIDENTIAL | JEREMY EDER54
0
:0
100
1:0
200
2:0
Ma+
# #3 #1 #0
3atencyBMicroecondC
#@tate lock impro"e determinim, red.ce itter
=ime 1@ec inter"alC
9.a" do ".e gu"! loo/ li/e 4
8/10/2019 201403 Tampa Performance3
74/99
RED HAT CONFIDENTIAL | JEREMY EDER5:
8/10/2019 201403 Tampa Performance3
75/99
9.a"?! .a##ening "o Tuned for RHEL& 4
8/10/2019 201403 Tampa Performance3
76/99
RED HAT CONFIDENTIAL | JEREMY EDER55
Tuned' +#da"e! for RHEL&
8/10/2019 201403 Tampa Performance3
77/99
RED HAT CONFIDENTIAL | JEREMY EDER5
'ntalled %y defa.lt
Tuned' +#da"e! for RHEL&
8/10/2019 201403 Tampa Performance3
78/99
RED HAT CONFIDENTIAL | JEREMY EDER5H
'ntalled %y defa.lt
Profile a.tomatically et %aed on intall typeD
&ektop8orktationD %alanced
Ser"erP#D thro.ghp.t@performance
Tuned' +#da"e! for RHEL&
i f i i %ili d i%ili
8/10/2019 201403 Tampa Performance3
79/99
RED HAT CONFIDENTIAL | JEREMY EDER0
e@written for maintaina%ility and e+teni%ility-
Tuned' +#da"e! for RHEL&
itt f i t i %ilit d t i%ilit
8/10/2019 201403 Tampa Performance3
80/99
RED HAT CONFIDENTIAL | JEREMY EDER1
e@written for maintaina%ility and e+teni%ility-
#onfig.ration conolidated to inglet.ned-conf file
Tuned' +#da"e! for RHEL&
itt f i t i %ilit d t i%ilit
8/10/2019 201403 Tampa Performance3
81/99
RED HAT CONFIDENTIAL | JEREMY EDER2
e@written for maintaina%ility and e+teni%ility-
#onfig.ration i now conolidated a inglet.ned-conf file
(ptional hookcallo.t capa%ility
Tuned' +#da"e! for RHEL&
itt f i t i %ilit d t i%ilit
8/10/2019 201403 Tampa Performance3
82/99
RED HAT CONFIDENTIAL | JEREMY EDER3
e@written for maintaina%ility and e+teni%ility-
#onfig.ration i now conolidated a inglet.ned-conf file
(ptional hookcallo.t capa%ility
9dd concept of 'nheritance .t likehttpd-confC
Tuned' +#da"e! for RHEL&
e written for maintaina%ility and e+teni%ility
8/10/2019 201403 Tampa Performance3
83/99
RED HAT CONFIDENTIAL | JEREMY EDER4
e@written for maintaina%ility and e+teni%ility-
#onfig.ration i now conolidated a inglet.ned-conf file
(ptional hookcallo.t capa%ility
9dd concept of 'nheritance .t likehttpd-confC
Profile .pdated for E5 feat.re andcharacteritic
Tuned' Profile In.eri"ance
8/10/2019 201403 Tampa Performance3
84/99
RED HAT CONFIDENTIAL | JEREMY EDER:
#hildren
Parent
[email protected]@performance
network@[email protected]
"irt.al@hot
%alanced
dektop
Tuned' Profile In.eri"ance
8/10/2019 201403 Tampa Performance3
85/99
RED HAT CONFIDENTIAL | JEREMY EDER
#hildren
Parent
[email protected]@performance
network@[email protected]
"irt.al@hot
%alanced
dektop
Qo.r@&BQo.r@8e% Qo.r@Middleware
9.a" do ".e gu"! loo/ li/e 4
8/10/2019 201403 Tampa Performance3
86/99
RED HAT CONFIDENTIAL | JEREMY EDER5
Tuna2le +ni"! -alanced ".roug.#u",#erformance ne"*or/,".roug.#u"
'nherit *rom/ote thro.ghp.t@performance
Tuned' T.roug.#u" Profile! , RHEL&
8/10/2019 201403 Tampa Performance3
87/99
RED HAT CONFIDENTIAL | JEREMY EDER
chedGminG gran.larityGn nanoecond a.to@caling 10000000
chedGwake.pGgran.larityGn nanoecond 3000000 1:000000
dirtyGratio Percent 20 40
dirtyG%ackgro.ndGratio Percent 10 10
wappine 8eight 1@100 0 10
'( Sched.ler Ele"atorC deadline
*ileytem Barrier Boolean Ena%led
#P$ o"ernor ondemand performance
&ik ead@ahead 6B 12 40H
&ia%le =P Boolean Ena%led
Energy Perf Bia normal performance
kernel-chedGmigrationGcotGn nanoecond :00000
minGperfGpct intelGptate onlyC Percent a.to@caling 100
tcpGrmem Byte a.to@caling Ma+L155521
tcpGwmem Byte a.to@caling Ma+L155521
.dpGmem Page a.to@caling Ma+L155521
Tunable +ni"! -alanced la"enc,#erformance ne"*or/,la"enc
'nherit *rom/ote la"enc,#erformance
chedGminG gran.larityGn nanoecond a.to@caling 10000000
Tuned' La"enc Profile! , RHEL&
8/10/2019 201403 Tampa Performance3
88/99
RED HAT CONFIDENTIAL | JEREMY EDERH
chedGwake.pGgran.larityGn nanoecond 3000000 10000000
dirtyGratio percent 20 10
dirtyG%ackgro.ndGratio percent 10 3
wappine 8eight 1@100 0 10
'( Sched.ler Ele"atorC deadline
*ileytem Barrier Boolean Ena%led
#P$ o"ernor ondemand performance
&ia%le =P Boolean /9 No Qe
#P$ #@State /9 ocked R 1
Energy Perf Bia normal performance
kernel-chedGmigrationGcotGn nanoecond /9 :000000
minGperfGpct intelGptate onlyC percent 100
net-core-%.yGread microecond :0
net-core-%.yGpoll microecond :0
net-ip"4-tcpGfatopen Boolean Ena%led
kernel-n.maG%alancing Boolean &ia%led
Tunable +ni"! ".roug.#u",#erformance )ir"ual,.o!" )ir"ual,gue!"
'nherit *rom/ote thro.ghp.t@performance
thro.ghp.t@performance
Tuned'
8/10/2019 201403 Tampa Performance3
89/99
RED HAT CONFIDENTIAL | JEREMY EDERH0
p p
chedGminG gran.larityGn nanoecond 10000000
chedGwake.pGgran.larityGn nanoecond 1:000000
dirtyGratio percent 40 30
dirtyG%ackgro.ndGratio percent 10 : 30
wappine 8eight 1@100 10
'( Sched.ler Ele"atorC
*ileytem Barrier Boolean
#P$ o"ernor performance
&ik ead@ahead Byte 40H
Energy Perf Bia performance
kernel-chedGmigrationGcotGn nanoecond :000000
minGperfGpct intelGptate onlyC percent 100
RHEL ="uned> #ac/age
Aailale profiles:
8/10/2019 201403 Tampa Performance3
90/99
RED HAT CONFIDENTIAL | JEREMY EDERH1
p alanced
des(top latenc;performance m;profile net
8/10/2019 201403 Tampa Performance3
91/99
RED HAT CONFIDENTIAL | JEREMY EDERH2
Po*er Managemen"
CP+ Tuning' C,!"a"e! 5idle !"a"e!6
P S i P f
8/10/2019 201403 Tampa Performance3
92/99
RED HAT CONFIDENTIAL | JEREMY EDERH3
0
:0
100
1:0
200
2:0
Ma+
# #3 #1 #0
#@tate 'mpact on Jitter
3atencyBMicro
econdC
=ime 1@ec inter"alC
Power Sa"ing Performant
8/10/2019 201403 Tampa Performance3
93/99
Tur2o!"a" !.o*! P(C,!"a"e! on In"el CP+!
t.r%otat in E-4, cp.power.til package or kernel@tool in *edoraE5
8/10/2019 201403 Tampa Performance3
94/99
RED HAT CONFIDENTIAL | JEREMY EDERH:
Def'u&t
p4 cor CPU c G.z *SC c$ c3 c c
62! 2673 26"" 562 $632 6 7262
$ $ 265! 363 26"" 36$3 6$5 6 7!6$"
2 2 2627 36" 26"" $6! 6 6 7625
3 3 $65 $65 26"" $62$ 6! 6$2 76!!
&'tenc8-perfor1'nce
p4 cor CPU c G.z *SC c$ c3 c c
6 363 267 $6 6 6 6
$ $ 6 363 267 $6 6 6 6
2 2 6 363 267 $6 6 6 6
3 3 6 363 267 $6 6 6 6
Profiling c#uidle and c#ufreB
*i+ed .ptream cp.idle regreion in J.ne
8/10/2019 201403 Tampa Performance3
95/99
RED HAT CONFIDENTIAL | JEREMY EDERH
*i+ed .ptream cp.idle regreion in J.ne
*.t.re #P$ .ch a awell add more #@tate,#,#H,#10
=.r%otat diplay a %it awkward for 40U core
&irection in f.t.re i to hrink the e+it@latency %etween#@tate
httpDwww-%reakage-org201211proceor@ma+Gctate@intelGidle@ma+Gctate@and@de"cp.GdmaGlatency
8/10/2019 201403 Tampa Performance3
96/99
RED HAT CONFIDENTIAL | JEREMY EDERH5
Ta/e,a*a!
Hel#ful +"ili"ie!
S.pporta%ility
/$M9
hwloc
/etworking
dropwatch
http://www.breakage.org/2012/11/processor-max_cstate-intel_idle-max_cstate-and-devcpu_dma_latency/http://www.breakage.org/2012/11/processor-max_cstate-intel_idle-max_cstate-and-devcpu_dma_latency/8/10/2019 201403 Tampa Performance3
97/99
RED HAT CONFIDENTIAL | JEREMY EDERH
[email protected]@tool
o
kd.mp
perf
pmic trace
ytat
ytemtap
trace@cmd
.til@lin.+@ng
'ntel P#M n.mactl
n.mad
n.matop 01-orgC
Power=.ning
cp.power.til C
kernel@tool 5C
powertop
t.na
t.ned
p
ethtool netniff@ng EPEC
tcpd.mp
wireharkthark
Storage
%lktrace
iotop
iotat
Hel#ful Lin/!
ed at ow atency Performance =.ning .ide
8/10/2019 201403 Tampa Performance3
98/99
RED HAT CONFIDENTIAL | JEREMY EDERHH
y g
(ptimi)ing E Performance %y =.ning '> 9ffinity
ed at Performance =.ning .ide
ed at 7irt.ali)ation =.ning .ide
S=9# /etwork '( S' *inteligent ow atency =.ning w67M
Perf
BlogD httpDwww-%reakage-orgor Reremyeder
https://access.redhat.com/knowledge/articles/2211538/10/2019 201403 Tampa Performance3
99/99
RED HAT CONFIDENTIAL | JEREMY EDER100
;ue!"ion!
http://www.breakage.org/https://perf.wiki.kernel.org/http://finteligent.net/pg/file/onxenterprise/read/51036http://www.stacresearch.com/niohttps://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Virtualization_Tuning_and_Optimization_Guide/index.htmlhttps://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Performance_Tuning_Guide/index.htmlhttps://access.redhat.com/knowledge/articles/216733https://access.redhat.com/knowledge/articles/221153