Upload
wyukawa
View
678
Download
3
Embed Size (px)
Citation preview
Elasticsearch, Nginx monitoringwithPrometheusandGrafana
2016/08/18Wataru Yukawa(@wyukawa)PrometheusLondonmeetup
WhoamI?
• DataEngineeratLINE• LINEmakesamessagingapplicationofthesamename,inadditiontootherrelatedservices
• ItisthemostpopularmessagingplatforminJapan
• IlikePremierLeague
Agenda
• browserlogcollectionsystem• monitorElasticsearch/Nginx
browser log collection system• FrontendengineerdevelopsJavaScriptbutit’sdifficulttodebugbecausetherearemanykindofbrowsers/devices.
• SowecreatebrowserlogcollectionsystemlikeGoogleAnalytics by using web beacon
• NginxlogsaresenttoESbyfluentd• Fluentd isanOSSlogcollectorlikelogstash,flume writteninruby.
Whatmiddlewareshouldbemonitored?
• Nginx• Elasticsearch• Fluentd– IwilltalkinPromCon2016
Nginxmonitoring• Usengx_http_stub_status_module
#monitoringlocation/___nginx_status{stub_statuson;allow127.0.0.1;denyall;
}
$curl-shttp://localhost/___nginx_statusActiveconnections:2serveracceptshandledrequests3010423010424242850Reading:0Writing:1Waiting:1
lifecycleofconnection
https://www.datadoghq.com/blog/how-to-monitor-nginx/
Nginxmonitoring• Usenginx_exporter– nginx_exporter usesngx_http_stub_status_moduleresult
#HELPnginx_connections_currentNumberofconnectionscurrentlyprocessedbynginx#TYPEnginx_connections_currentgaugenginx_connections_current{state="active"}2nginx_connections_current{state="reading"}0nginx_connections_current{state="waiting"}1nginx_connections_current{state="writing"}1#HELPnginx_connections_processed_totalNumberofconnectionsprocessedbynginx#TYPEnginx_connections_processed_totalcounternginx_connections_processed_total{stage="accepted"}300966nginx_connections_processed_total{stage="any"}4.241939e+06nginx_connections_processed_total{stage="handled"}300966
nginxdashboard
requestconnection
irate(nginx_connections_processed_total{...,stage="any"}[1m])
Elasticsearch monitoring
• Useelasticsearch_exporter• UseMarvel
Elasticsearch dashboard
javaheap
https://github.com/elastic/elasticsearch/issues/18635
Myfeeling
• Prometheus’squeryisreallypowerfulespeciallywhenmetrictypeiscounter
• Wecanuserate/iratefunction.• nginx_exporter and elasticsearch_exporter areconvienient– easytosetup
• PrometheusandGrafana areaperfectcombination
References
• https://github.com/discordianfish/nginx_exporter
• https://github.com/ewr/elasticsearch_exporter