Upload
anis-oneal
View
231
Download
0
Embed Size (px)
DESCRIPTION
Background Modern Internet services are often implemented as complex, large-scale distributed systems. These applications are constructed from collections of software modules that may be developed by different teams, perhaps in different programming language, and could span many thousand of machines across multiple physical facilities
Citation preview
Dapper, a Large-Dapper, a Large-Scale Distributed Scale Distributed System Tracing System Tracing InfrastructureInfrastructureGoogle Technical Report, 2010Author: B.H.Sigelman, L.A.Barroso, M.Burrows, P.Stephenson, M.Plakal, D.Beaver, S.Jaspan, C.ShanbhagPresenter: Lei Jinjiang
BackgroundBackground
Modern Internet services are often implemented as complex, large-scale distributed systems. These applications are constructed from collections of software modules that may be developed by different teams, perhaps in different programming language, and could span many thousand of machines across multiple physical facilities
BackgroundBackground
Imagine a single search request coursing through Google’s massive infrastructure. A single request can run across thousands of machines and involve hundred of different subsystems. And oh by the way, you are processing more requests per second than any other system in the world.
ProblemProblem
• How do you debug such a system?• How do you figure out where the problem are?• How do you determine if programmers are
coding correctly?• How do you keep sensitive data secret and
safe?• How do ensure products don’t use more
resources than the are assigned?• How do you store all the data?• How do you make use of it?
That is where dapper comes in!That is where dapper comes in!
DapperDapper
Dapper is Google's tracing system and it was originally created to understand the system behavior from a search request. Now Google's production clusters generate more than 1 terabyte of sampled trace data per day.
Requirements and Design Requirements and Design GoalsGoals
• Requirements: (1) Ubiquitous deployment (2) Continuous monitoring• Design Goals: (1) Low overhead (2) Application-level transparency (3) Scalability
Distributed Tracing in DapperDistributed Tracing in Dapper
Two class of solutions:Black-box vs. annotation-based
Trace trees and spansTrace trees and spans
The causal and temporal relationships between five spans in a Dapper trace tree
Trees and SpansTrees and Spans
A detailed view of a single span from last Figure
Instrumentation pointsInstrumentation points• When a thread handles a traced
control path, Dapper attaches a trace context to thread-local storage.
• Most Google developers use a common control flow library to construct callbacks. Dapper ensures that all such callback store the trace context
CallbackCallback
In computer programming, a callback is a reference to executable code, or a piece of executable code, that is passed as an argument to other code. This allows a lower-level software layer to call a subroutine (or function) defined in a higher-level layer.
AnnotationsAnnotations// C++:const string& request = ...;if (HitCache()) TRACEPRINTF("cache hit for %s", request.c_str());else TRACEPRINTF("cache miss for %s", request.c_str());
// Java:Tracer t = Tracer.getCurrentTracer();String request = ...;if (hitCache()) t.record("cache hit for " + request);else t.record("cache miss for " + request);
SamplingSampling
Low overhead was a key design goal for Dapper, since service operators would be understandably reluctant to deploy a new tool of yet unproven value if it had any significant impact on performance…Therefore, … , we further control overhead by recording only a fraction of all traces.
Trace collectionTrace collection
Out-of-band trace collectionOut-of-band trace collection
• Firstly, the in-band-Dapper trace data would dwarf the application data and bias the results of subsequent analyses.
• Secondly, many middleware systems which return a result to their caller before all of their own backend have returned a final result.
Security and privacy Security and privacy considerationsconsiderations
Production coverageProduction coverage• Given how ubiquitous Dapper-instrumented
libraries are, we estimate that nearly every Google production process supports tracing.
• There are cases where Dapper is unable to follow the control path correctly. These typically stem from the use of non-standard control-flow primitives.
• Dapper tracing can be turned off as a production safety measure.
Use of trace annotationsUse of trace annotations• Currently, 70% of all Dapper spans and
90% of all Dapper traces have at least one application-specified annotation.
• 41 Java and 68 C++ applications have added custom application annotations in order to better understand intra-span activity in their sevices.
Trace collection overheadTrace collection overhead
• The daemon never uses more than 0.3% of one core of a production machine during collection, and has a very small memory footprint.
• Restrict the Dapper daemon to the lowest possible priority in the kernel scheduler.
• 426 bytes/span, less than 0.01% of the network traffic in Google’s production environment
Process count(per host)
Data Rate(per process)
Daemon CPU Usage(single CPU core)
25 10K/sec 0.125%
10 200K/sec 0.267%
50 2K/sec 0.130%
CPU resource usage for the Dapper daemon during load testing
Trace collection overheadTrace collection overheadSampling frequency
Avg. Latency(% change)
Avg. Throughput(% change)
1/1 16.3% -1.48%
1/2 9.40% -0.73%
1/4 6.38% -0.30%
1/8 4.12% -0.23%
1/16 2.12% -0.08%
1/1024 -0.20% -0.06%
The effect of different [non-adaptive] Dapper sampling frequencies on the latency and throughput of a Web search cluster. The experimental errors for these latency and throughput measurements are 2.5% and 0.15% respectively.
Adaptive sampling• Lower traffic workloads may miss important events at
such low sampling rate(1/1024).
• Workloads with low traffic automatically increase their sampling rate while those with very high traffic will lower it so that overheads remain under control.
• Reliability …
Additional sampling during Additional sampling during collectioncollection
• The dapper team also need to control the total size of data written to its central repositories, and thus we incorporate a second round of sampling for that purpose.
• For each span seen in the collection system, we hash the associate trace id as a scalar z, where 0≤z≤1. If z is less than our collection sampling coefficient, we keep the span and write it to the Bigtable. Otherwise, we discard it.
The Dapper Depot APIThe Dapper Depot API• Access by trace id• Bulk access: Access to billions of Dapper in
parallel• Indexed access: The index maps from commonly
requested trace feature(host machine, service names) to distinct dapper traces.
User interfaceUser interface
ExperiencesExperiences• Using Dapper during development (Integration with exception monitoring)• Addressing long tail latency• Inferring service dependencies• Network usage of differ services• Layered and Shared Storage Systems (e.g.
GFS)
Other Lessons LearnedOther Lessons Learned• Coalescing effect• Tracing batch workloads• Finding a root cause• Logging kernel-level information