27 au 29 mars 2013
Monitoring Open Source pour Java avec JmxTrans, Graphite et Nagios
Wednesday, March 27, 13
Speaker
Cyrille Le Clerc
@cyrilleleclerc
Open Source
CTO
Fort Traffic
Cloud, DevOps
Architect
Wednesday, March 27, 13
Speaker
Henri Gomez
@hgomez
Open Source
Intégration Continue
DevOps
Architecte CI
Wednesday, March 27, 13
Vous ?
Dev ? Qa ? Ops ?Expérience monitoring ?
Wednesday, March 27, 13
27 au 29 mars 2013
Agenda
Wednesday, March 27, 13
Agenda
• L’application
• Le monitoring à réaliser
• Instrumenter une webapp Java avec JMX
• Monitorer
• Grapher avec Graphite
• Alerter avec Seyren (et Nagios)
• Monitoring & Continuous Delivery
• Déploiement de Graphite
• Conclusion
Wednesday, March 27, 13
Votre mission, si toutefois vous l’acceptez ...
Mettre en place le monitoring technique et métier
e-commerce SOS Cocktails
http://www.flickr.com/photos/23791504@N08/2607814349
Wednesday, March 27, 13
SOS CocktailDémo
Wednesday, March 27, 13
27 au 29 mars 2013
Les indicateurs à monitorer
Pourquoi Monitorer ?Quoi ?
Wednesday, March 27, 13
Les indicateurs à monitorer
Wednesday, March 27, 13
Les Indicateurs à monitorer
• Operating System -> Sysload
• JVM -> GC duration
• Tomcat -> activeSessions (active visitors)
• Application -> sales revenue & items
Wednesday, March 27, 13
Open Source Monitoring for Java
eCommerce Web Site Graphite
Seyren
Dashboard - wiki
Nagios
(1)(2) (3)
(4)
(5)
Wednesday, March 27, 13
27 au 29 mars 2013
JMX & Métriques
Wednesday, March 27, 13
Pourquoi JMX ?
• JMX vs. pages web vs. logs vs. ...
• Simple et sécurisé
• Exposable vers pages web et les logs
• Des alternatives à JMX
Wednesday, March 27, 13
JMX vs. logs vs. pages web vs. ...
• Logs de monitoring
• Parser des Go de fichiers texte ?
• Détail par invocation vs. moyenne par minute
• Pages web
• Historisation ?
• Applications en clusters ?
• Sécurité ?
Wednesday, March 27, 13
JMX avec Spring<beans ...> ... <context:mbean-‐export/> ...</beans>
@ManagedResource("cocktail:name=ShoppingCartController,type=...")class ShoppingCartController { final AtomicInteger revenueInCentsCounter = new AtomicInteger();
@ManagedAttribute public int getRevenueInCentsCounter() { return revenueInCentsCounter.get(); } void purchase(...){ ... revenueInCentsCounter.addAndGet(priceInCents); }
Wednesday, March 27, 13
JMX
Wednesday, March 27, 13
JMX & JSP
$ curl http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-‐sales-‐metrics.txt.jspEpoch SalesRevenueInCentsCounter SalesItemsCounter SalesOrdersCounter1363217376 4500 7 2
curl http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/...{ "Epoch":1363217653, "SalesRevenueInCentsCounter":4050, "SalesItemsCounter":7, "SalesOrdersCounter":2}
http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp
Powered by JMX!
Human readable et script friendlyAttention sécurité !
Wednesday, March 27, 13
JMX & JSP
$ curl http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/jmx-‐sales-‐metrics.txt.jspEpoch SalesRevenueInCentsCounter SalesItemsCounter SalesOrdersCounter1363217376 4500 7 2
curl http://demo-‐cocktail.jmxtrans.cloudbees.net/5ADrUWr5/...{ "Epoch":1363217653, "SalesRevenueInCentsCounter":4050, "SalesItemsCounter":7, "SalesOrdersCounter":2}
http://demo-cocktail.jmxtrans.cloudbees.net/jmx.jsp
Powered by JMX!
Human readable et script friendlyAttention sécurité !
/5ADrUWr5/jmx-sales-metrics.jsp
Wednesday, March 27, 13
JMX & JSPPowered by JMX!
Wednesday, March 27, 13
Alternatives à JMX
Wednesday, March 27, 13
Alternatives à JMX
class Shopping
CartController
{
@Monitor(name
="revenueInCen
ts", type=COUN
TER)
final AtomicI
nteger revenue
InCentsCounter
= new AtomicI
nteger();
void p
urchase(...){
...
revenueInCe
ntsCounter.add
AndGet(priceIn
Cents);
}
Servo
Wednesday, March 27, 13
Alternatives à JMX
class Shopping
CartController
{
@Monitor(name
="revenueInCen
ts", type=COUN
TER)
final AtomicI
nteger revenue
InCentsCounter
= new AtomicI
nteger();
void p
urchase(...){
...
revenueInCe
ntsCounter.add
AndGet(priceIn
Cents);
}
Servo
class ShoppingCartController { final Counter revenueInCentsCounter = Metrics.newCounter( ShoppingCartController, "revenue-‐in-‐cents"); void purchase(...){ ... revenueInCentsCounter.inc(priceInCents); }
Metrics
Wednesday, March 27, 13
JMX
•OS: Sysload•JVM: Garbage Collector•Tomcat: activeSessions •Application: Web Site Revenue
Wednesday, March 27, 13
27 au 29 mars 2013
jmxtrans & embedded-jmxtrans
Wednesday, March 27, 13
jmxtrans - standalone
jmxtrans - standalone
ecommerceTomcat
Graphite
accountingTomcat
logisticsTomcat
...
Wednesday, March 27, 13
embedded-jmxtrans
embedded-jmxtrans
logistics
Tomcat
Graphite
ecommerce
Tomcat
accounting
Tomcat
Librato
...
batch
Wednesday, March 27, 13
standalone vs. embedded
standalone embedded
Packaging
Modèledaemon externe, raccordement manuel des applications & des
indicateurs
agent embarqué, applications & indicateurs auto-raccordés /
autonomes
Curseur OPS -♦----------- DEV OPS ----------♦-- DEV
Use case applis “peu modifiables”, middlewares, curseur OPS
cloud, batch, curseur DEV/QA
Wednesday, March 27, 13
jmxtrans & embedded-jmxtrans
SOS Cocktail Graphite
embedded
Wednesday, March 27, 13
embedded-jmxtrans configuration
Wednesday, March 27, 13
embedded-jmxtrans configuration
<dependency> <groupId>org.jmxtrans.embedded</groupId> <artifactId>embedded-‐jmxtrans</artifactId> <version>1.0.5</version></dependency>
Wednesday, March 27, 13
embedded-jmxtrans configuration
<dependency> <groupId>org.jmxtrans.embedded</groupId> <artifactId>embedded-‐jmxtrans</artifactId> <version>1.0.5</version></dependency>
<beans ... xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded" xsi:schemaLocation="... http://www.jmxtrans.org/schema/embedded ...”>
<jmxtrans:jmxtrans> <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration> ... </jmxtrans:jmxtrans></beans>
Wednesday, March 27, 13
embedded-jmxtrans configuration
<dependency> <groupId>org.jmxtrans.embedded</groupId> <artifactId>embedded-‐jmxtrans</artifactId> <version>1.0.5</version></dependency>
<beans ... xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded" xsi:schemaLocation="... http://www.jmxtrans.org/schema/embedded ...”>
<jmxtrans:jmxtrans> <jmxtrans:configuration>classpath:jmxtrans.json</jmxtrans:configuration> ... </jmxtrans:jmxtrans></beans>
{ "queries": [ { "objectName": "cocktail:name=ShoppingCartController", "resultAlias": "sales", "attributes": ["SalesRevenueInCentsCounter"] }, ], "outputWriters": [ { "@class": "org.jmxtrans.embedded.output.GraphiteWriter", "settings": { "host": "${graphite.host:localhost}", "port": "${graphite.port:2003}," "enabled": "${graphite.enabled:true}" } } ]}
Wednesday, March 27, 13
jmxtrans
•OS: Sysload•JVM: Garbage Collector•Tomcat: activeSessions •Application: Web Site Revenue
Wednesday, March 27, 13
27 au 29 mars 2013
Graphite
Wednesday, March 27, 13
Graphite
• BD de Time Series et solution de Graphe
• Utilisé par des grands du Web
• Similaires
Simplicité et Self Service
Wednesday, March 27, 13
Ecriture / Alimentation
• Création automatique des métriques
• Protocoles socket “plain text” & sérialisation Python Pickle
• Précision et durée de stockage par configuration
echo "devoxx.happyCounter 1.2 1364338989" | nc localhost 2003
[sales_1min_for_15days_5min_for_1year]pattern = ^sales\.retentions = 60s:30d,300s:365d 1.7MB
Wednesday, March 27, 13
Lecture / Consommation
• Composer de graphe & URL API
• Tout le monde peut créer ses graphes
• Fonctions Riches• sum, scale, derivative, timeshift, deviation, filter, ...
• Exports variés : png, svg, csv, json, raw text
Wednesday, March 27, 13
27 au 29 mars 2013
Graphes et monitoring
Wednesday, March 27, 13
Demo Graphite sur le Cloud
• http://hostedgraphite.com/
• Graphite as a Service• Pas de ‘tracas’ d’installation
• addons: Tasseo
• No-OPS
1
2
3
email: [email protected]
password: devoxx2013
Wednesday, March 27, 13
Les métriques
• Exemples: activeRequests, dataSource.activeConnection, ...
• Information exploitable immédiatement
Gauge Ever Increasing Counter
• Exemples: requestsCount, revenue, ...
• Conversion nécessaire en “par minute”
Wednesday, March 27, 13
Des graphes et des formules
?
Ever Increasing Counter Revenue per Second
Wednesday, March 27, 13
Des graphes et des formules
?
Ever Increasing Counter Revenue per Second
derivative()
Wednesday, March 27, 13
Des graphes et des formules
?
Revenue per HourPer Second -> Per Hour
Wednesday, March 27, 13
Des graphes et des formules
?
Revenue per Hour
summarize()
Per Second -> Per Hour
Wednesday, March 27, 13
Des graphes et des formules
2 servers
?
Total Revenue per Hour
Wednesday, March 27, 13
Des graphes et des formules
2 servers
?
Total Revenue per Hour
sumSeries()
Wednesday, March 27, 13
Des graphes et des formules
server restart
?
Ignore reset to zero
Wednesday, March 27, 13
Des graphes et des formules
server restart
?
Ignore reset to zero
nonNegativeDerivative()
Wednesday, March 27, 13
Des graphes et des formules
Compare to last week
?
Wednesday, March 27, 13
Des graphes et des formules
Compare to last week
?timeShift()
Wednesday, March 27, 13
Des graphes et des formules
•OS: Sysload•JVM: Garbage Collector•Tomcat: activeSessions •Application: Web Site Revenue
Wednesday, March 27, 13
Dashoards Wikidashboard.prod.md# Sales
<table><tr> <td> <img src="http://graphite.prod.mycompany/graphite/render/?width=400&height=300&title=Revenue&xFormat=%25a%20%25d%25p&vtitle=USD&lineWidth=3&from=-‐5days&target=alias(scale(summarize(sumSeries(nonNegativeDerivative(edu.servers.*.sales.revenueInCentsCounter))%2C%221h%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)..."> </td> <td> <img src="http://graphite.prod.mycompany/graphite/render/?..."> </td><tr></table>
https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/Dashboard-PROD
Wiki
Wednesday, March 27, 13
Dashoards by Email
Envoyer les graphes par email
Wednesday, March 27, 13
27 au 29 mars 2013
L’alerting
Intégrer Graphite à de l’alertingURL ApiSeyrenNagios
Wednesday, March 27, 13
Intégrer Graphite à l’alerting
• Utiliser l’URL API
• Format rawText ou JSON
• Pattern /render?from=-‐11minutes&until=-‐1minutes&format=raw&target=****
$ curl "http://graphite.exemple.com/render?from=-‐11minutes&until=-‐1minutes&format=raw& target=keepLastValue(servers.cloudbees.jvm.os.SystemLoadAverage)"
my-‐metric,1363225680,1363226340,60|0.03,0.01,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0
Exemple
Wednesday, March 27, 13
Alerting - Seyren
• Alerting Dashboard for Graphite
• Open Source
• Java .war + MongoDB
• Alertes email, PagerDuty, ...
Wednesday, March 27, 13
Alerting - Seyren
•OS: Sysload•JVM: Garbage Collector•Tomcat: activeSessions •Application: Web Site Revenue
http://seyren.jmxtrans.cloudbees.net/
Wednesday, March 27, 13
Nagios
• Infrastructure Monitoring
• Open Source
• Standard de facto en entreprise
• check_graphite plugin
• Pierre-Yves Ritschard
• Jason Dixon / obfuscurity
Wednesday, March 27, 13
27 au 29 mars 2013
Monitoring & Continuous Delivery
Wednesday, March 27, 13
Monitoring & Continuous Delivery
• Livrer le monitoring avec l’application
• Les indicateurs
• Les dashboards
• Builder en continue les configs par environnement
• Require: topologie des environnements intégrées au build
• Déploiement sur un repo (Maven, ...)
• Déployer en automatique le monitoring
Wednesday, March 27, 13
Monitoring & Continuous Delivery
src/main/graphite/dashboard.template.md
Cocktail App Dashboards -‐ ${environment} -‐ ${version}# Sales
<table><tr> <td> <img src="${graphite.baseUrl}/render/?width=400&height=300&title=Revenue&xFormat=%25a%20%25d%25p&vtitle=USD&lineWidth=3&from=-‐5days&target=alias(scale(summarize(sumSeries(nonNegativeDerivative(edu.servers.*.sales.revenueInCentsCounter))%2C%221h%22)%2C0.01)%2C%22Revenue%20per%20Hour%22)..."> </td> <td> <img src="${graphite.baseUrl}/render/?..."> </td><tr></table>
StringTemplate
${variable}
Dashboard template
Wednesday, March 27, 13
Monitoring & Continuous Delivery
dashboard.template.md
dashboard-generator.groovy
PROD-env-topology.json
pom.xml<gmaven-plugin>
dashboard.PROD.md
QA-env-topology.json
...
dashboard.QA.md
...
https://github.com/jmxtrans/embedded-jmxtrans-samples/wiki/Monitoring-Continuous-Deployment
repository
Monitoring dashboards Continuous Integration
Wednesday, March 27, 13
27 au 29 mars 2013
Déploiement de Graphite
Wednesday, March 27, 13
Déploiement de Graphite
Wednesday, March 27, 13
27 au 29 mars 2013
Cadeau bonus !
Wednesday, March 27, 13
Promo Code
• http://hostedgraphite.com/
• 60 days trial with “DEVOXX” promo code
(1) Signup
(2) Enter Promo Code “DEVOXX”
help: [email protected] (3) Promo Code “DEVOXX”
Wednesday, March 27, 13
27 au 29 mars 2013
Conclusion
Wednesday, March 27, 13
Conclusion
• Monitoring technique et métier
• Monitoring intégrée au Continuous Delivery de l’appli
• L’approche self-service transforme le monitoring
• Solutions Open Source disponibles
Monitoring no longer sucks in Java!http://lusislog.blogspot.fr/2011/06/why-monitoring-sucks.html
Wednesday, March 27, 13
http://demo-cocktail.jmxtrans.cloudbees.net/ http://jmxtrans.org/
Wednesday, March 27, 13