Distributed Tracing

Dappergoogle

Zipkintwitter

What's this?

– wikipedia.org `Tracing(software)`

“In software engineering, tracing involves a specialized use of logging to record information

about a program's execution. ”

What is slow?

Who did wrong?

How?

Possible solution• Blackbox

• application-agnostic

• low correctness

• Unique identifier

• application-intrusive

• instrumentation for various transports

Pre-Order Traversal

• Log parent entering

• Log all children

• Log parent leaving

Zipkin Example

Some infrastructure

• HTTP: X-Header

• Thrift: add tracer

• RPC: god-mode

• ...

Clock skewcentral problem of distributed system

Things worth mentioned

• sampling

• reporting

• aggregate analysis

Zipkin

Annotationstring data associated with particular timestamp,

service, host, ...

Annotation Types• cs: client start

• sr: server receive

• ss: server send

• cr: client receive

• other: further insight of service

SpanA set of annotations, represents one specific method call.

Has name, id.

TraceA set of span all associated to the same request

Identifiers

• Trace ID

• Span ID

• Parent ID

Big Brother Bird• X-B3-TraceId: 64 encoeded bits

• X-B3-SpanId: 64 encoded bits

• X-B3-ParentSpanId: 64 encoded bits

• X-B3-Sampled: Boolean (1/0

• X-B3-Flags: a Long

Architecture

Some adjustment

• scribe -> fluentd + fluentd-plugin-scribe

• fiangle -> http-based intrakit

• serialization & deserialization annotations

Q&A

Thanks@soasme