Upload
kawamuray
View
8.455
Download
7
Embed Size (px)
Citation preview
Monitoring Kafka w/ PrometheusYuto Kawamura(kawamuray)
About me● Software Engineer @ LINE corp
○ Develop & operate Apache HBase clusters○ Design and implement data flow between services with ♥ to Apache Kafka
● Recent works○ Playing with Apache Kafka and Kafka Streams
■ https://issues.apache.org/jira/browse/KAFKA-3471?jql=project%20%3D%20KAFKA%
20AND%20assignee%20in%20(kawamuray)%20OR%20reporter%20in%20(kawamuray)
● Past works○ Docker + Checkpoint Restore @ CoreOS meetup http://www.slideshare.
net/kawamuray/coreos-meetup
○ Norikraでアプリログを集計してリアルタイムエラー通知 @ Norikra Meetup #1 http://www.slideshare.net/kawamuray/norikra-meetup
○ Student @ Google Summer of Code 2013, 2014● https://github.com/kawamuray
How are we(our team) using Prometheus?● To monitor most of our middleware, clients on Java applications
○ Kafka clusters○ HBase clusters○ Kafka clients - producer and consumer○ Stream Processing jobs
Overall Architecture
Grafana
Prometheus
HBase clusterHBase
clusterKafka cluster
PrometheusPrometheus
Prometheus(Federation)
PrometheusPrometheus
Prometheus
YARN Application
Pushgateway
Dashboard
Direct query
Why Prometheus?● Inhouse monitoring tool wasn’t enough for large-scale + high resolution metrics
collection● Good data model
○ Genuine metric identifier + attributes as labels
■ http_requests_total{code="200",handler="prometheus",instance="localhost:9090",job="prometheus",method="get"}
● Scalable by nature● Simple philosophy
○ Metrics exposure interface: GET /metrics => Text Protocol○ Monolithic server
● Flexible but easy PromQL○ Derive aggregated metrics by composing existing metrics○ E.g, Sum of TX bps / second of entire cluster
■ sum(rate(node_network_receive_bytes{cluster="cluster-A",device="eth0"}[30s]) * 8)
Deployment● Launch
○ Official Docker image: https://hub.docker.com/r/prom/prometheus/
○ Ansible for dynamic prometheus.yml generation based on inventory and container management
● Machine spec○ 2.40GHz * 24 CPUs○ 192GB RAM○ 6 SSDs○ Single SSD / Single Prometheus instance
○ Overkill? => Obviously. Reused existing unused servers. You must don’t need this crazy spec just to use it.
Kafka monitoring w/ Prometheus overview
Kafka broker
Kafka client in Java application
YARN ResourceManager
Stream Processing jobs on YARN
Prometheus Server
Pushgateway
Jmx exporter
Prometheus Java
library+ Servlet
JSON exporter
Kafka consumer
group exporter
Monitoring Kafka brokers - jmx_exporter● https://github.com/prometheus/jmx_exporter● Run as standalone process(no -javaagent)
○ Just in order to avoid cumbersome rolling restart○ Maybe turn into use javaagent on next opportunity of rolling restart :p
● With very complicated config.yml○ https://gist.github.com/kawamuray/25136a9ab22b1cb992e435e0ea67eb06
● Colocate one instance per broker on the same host
Monitoring Kafka producer on Java application - prometheus_simpleclient● https://github.com/prometheus/client_java● Official Java client library
prometheus_simpleclient - Basic usageprivate static final Counter queueOutCounter = Counter.build() .namespace("kafka_streams") // Namespace(= Application prefix?) .name("process_count") // Metric name .help("Process calls count") // Metric description .labelNames("processor", "topic") // Declare labels .register(); // Register to CollectorRegistry.defaultRegistry (default, global registry)
...queueOutCounter.labels("Processor-A", "topic-T").inc(); // Increment counter with labelsqueueOutCounter.labels("Processor-B", "topic-P").inc(2.0);
=> kafka_streams_process_count{processor="Processor-A",topic="topic-T"} 1.0kafka_streams_process_count{processor="Processor-B",topic="topic-P"} 2.0
Exposing Java application metrics● Through servlet
○ io.prometheus.client.exporter.MetricsServlet from simpleclient_servlet
● Add an entry to web.xml or embedded jetty ..
Server server = new Server(METRICS_PORT); ServletContextHandler context = new ServletContextHandler(); context.setContextPath("/"); server.setHandler(context); context.addServlet(new ServletHolder(new MetricsServlet()), "/metrics"); server.start();
Monitoring Kafka producer on Java application - prometheus_simpleclient● Primitive types:
○ Counter, Gauge, Histogram, Summary● Kafka’s MetricsRerpoter interface gives KafkaMetrics instance● How to expose the value?● => Implement proxy metric type which implements
SimpleCollector public class PrometheusMetricsReporter implements MetricsReporter {... private void registerMetric(KafkaMetric kafkaMetric) { ... KafkaMetricProxy.build() .namespace(“kafka”) .name(fqn) .help("Help: " + metricName.description()) .labelNames(labelNames) .register(); ... }...}
public class KafkaMetricProxy extends SimpleCollector<KafkaMetricProxy.Child> { public static class Builder extends SimpleCollector.Builder<Builder, KafkaMetricProxy> { @Override public KafkaMetricProxy create() { return new KafkaMetricProxy(this); } }
KafkaMetricProxy(Builder b) { super(b); }
...
@Override public List<MetricFamilySamples> collect() { List<MetricFamilySamples.Sample> samples = new ArrayList<>(); for (Map.Entry<List<String>, Child> entry : children.entrySet()) { List<String> labels = entry.getKey(); Child child = entry.getValue(); samples.add(new Sample(fullname, labelNames, labels, child.getValue())); } return Collections.singletonList(new MetricFamilySamples(fullname, Type.GAUGE, help, samples)); }}
Monitoring YARN jobs - json_exporter● https://github.com/kawamuray/prometheus-json-exporter
○ Can export value from JSON by specifying the value as JSONPath
● http://<rm http address:port>/ws/v1/cluster/apps○ https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-
site/ResourceManagerRest.html#Cluster_Applications_API○ https://gist.github.com/kawamuray/c07b03de82bf6ddbdae6508e27d3fb4d
json_exporter- name: yarn_application type: object path: $.apps.app[*]?(@.state == "RUNNING") labels: application: $.id phase: beta values: alive: 1 elapsed_time: $.elapsedTime allocated_mb: $.allocatedMB...
{"apps":{"app":[ { "id": "application_1234_0001", "state": "RUNNING", "elapsedTime": 25196, "allocatedMB": 1024, ... },... }}
+
yarn_application_alive{application="application_1326815542473_0001",phase="beta"} 1yarn_application_elapsed_time{application="application_1326815542473_0001",phase="beta"} 25196yarn_application_allocated_mb{application="application_1326815542473_0001",phase="beta"} 1024
Important configurations● -storage.local.retention(default: 15 days)
○ TTL for collected values● -storage.local.memory-chunks(default: 1M)
○ Practically controls memory allocation of Prometheus instance○ Lower value can cause ingestion throttling(metric loss)
● -storage.local.max-chunks-to-persist(default: 512K)○ Lower value can cause ingestion throttling likewise○ https://prometheus.io/docs/operating/storage/#persistence-pressure-and-rushed-mode
○ > Equally important, especially if writing to a spinning disk, is raising the value for the storage.
local.max-chunks-to-persist flag. As a rule of thumb, keep it around 50% of the storage.local.memory-chunks value.
● -query.staleness-delta(default: 5mins)○ Resolution to detect lost metrics○ Could lead weird behavior on Prometheus WebUI
Query tips - label_replace function● It’s quite common that two metrics has different label sets
○ E.g, server side metric and client side metrics● Say have metrics like:
○ kafka_log_logendoffset{cluster="cluster-A",instance="HOST:PORT",job="kafka",partition="1234",topic="topic-A"}
● Introduce new label from existing label○ label_replace(..., "host", "$1", "instance", "^([^:]+):.*")
○ => kafka_log_logendoffset{...,instance=”HOST:PORT”,host=”HOST”}
● Rewrite existing label with new value○ label_replace(..., "instance", "$1", "instance", "^([^:]+):.*")
○ => kafka_log_logendoffset{...,instance=”HOST”}
● Even possible to rewrite metric name… :D○ label_replace(kafka_log_logendoffset, "__name__", "foobar", "__name__", ".*")
○ => foobar{...}
Points to improve● Service discovery
○ It’s too cumbersome to configure server list and exporter list statically○ Pushgateway?
■ > The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose
their metrics to Prometheus - https://github.com/prometheus/pushgateway/blob/master/README.md#prometheus-pushgateway-
○ file_sd_config? https://prometheus.io/docs/operating/configuration/#<file_sd_config>
■ > It reads a set of files containing a list of zero or more <target_group>s. Changes to all defined files are detected via disk watches and applied immediately.
● Local time support :(○ They don’t like TZ other than UTC; making sense though: https://prometheus.
io/docs/introduction/faq/#can-i-change-the-timezone?-why-is-everything-in-utc?○ https://github.com/prometheus/prometheus/issues/500#issuecomment-167560093○ Still might possible to introduce toggle on view
Conclusion● Data model is very intuitive● PromQL is very powerful and relatively easy
○ Helps you find out important metrics from hundreds of metrics
● Few pitfalls needs to be avoid w/ tuning configurations○ memory-chunks, query.staleness-detla…
● Building exporter is reasonably easy○ Officially supported lot’s of languages…○ /metrics is the only interface
Questions?
End of Presentation
Metrics naming● APPLICATIONPREFIX_METRICNAME
○ https://prometheus.io/docs/practices/naming/#metric-names○ kafka_producer_request_rate○ http_request_duration
● Fully utilize labels○ x: kafka_network_request_duration_milliseconds_{max,min,mean}○ o: kafka_network_request_duration_milliseconds{“aggregation”=”max|min|mean”}
○ Compare all min/max/mean in single graph: kafka_network_request_duration_milliseconds{instance=”HOSTA”}
○ Much flexible than using static name
Alerting● Not using Alert Manager● Inhouse monitoring tool has alerting capability
○ Has user directory of alerting target○ Has known expression to configure alerting
○ Tool unification is important and should be respected as possible
● Then?○ Built a tool to mirror metrics from Prometheus to inhouse
monitoring tool○ Setup alert on inhouse monitoring tool
/api/v1/query?query=sum(kafka_stream_process_calls_rate{client_id=~"CLIENT_ID.*"}) by (instance)
{ "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "instance": "HOST_A:PORT" }, "value": [ 1465819064.067, "82317.10280584119" ] }, { "metric": { "instance": "HOST_B:PORT" }, "value": [ 1465819064.067, "81379.73499610288" ] }, ] }}
public class KafkaMetricProxy extends SimpleCollector<KafkaMetricProxy.Child> {... public static class Child { private KafkaMetric kafkaMetric;
public void setKafkaMetric(KafkaMetric kafkaMetric) { this.kafkaMetric = kafkaMetric; }
double getValue() { return kafkaMetric == null ? 0 : kafkaMetric.value(); } }
@Override protected Child newChild() { return new Child(); }...}
Monitoring Kafka consumer offset - kafka_consumer_group_exporter● https://github.com/kawamuray/prometheus-kafka-consumer-group-exporter● Exports some metrics WRT Kafka consumer group by executing kafka-
consumer-groups.sh command(bundled to Kafka)● Specific exporter for specific use● Would better being familiar with your favorite exporter framework
○ Raw use of official prometheus package: https://github.com/prometheus/client_golang/tree/master/prometheus
○ Mine: https://github.com/kawamuray/prometheus-exporter-harness
Query tips - Product set● Calculated result of more than two metrics results product set
● metric_A{cluster=”A or B”}● metric_B{cluster=”A or B”,instance=”a or b or c”}● metric_A / metric_B● => {}● metric_A / sum(metric_B) by (cluster)● => {cluster=”A or B”}● x: metric_A{cluster=”A”} - sum(metric_B{cluster=”A”}) by (cluster)● o: metric_A{cluster=”A”} - sum(metric_B) by (cluster) => Same result!