Beringei: Facebook's Open Source, In-Memory Time Series Database (TSDB)

Santa Clara, California | April 24th – 27th, 2017

3

Overview

• Why build Beringei?

• Overview of the system

• Comparison with Prometheus

• How to use Beringei in your project

4

Why build Beringei?

• In early 2013, we realized our disk-backed time series storage would not scale to future read loads• 90th percentile query time was > 2s

• Our users were self-censoring queries to return smaller data sets

• We decided to utilize an in-memory store to drive large volume analysis• Needs to be real-time and highly available

• Drive dense visualizations and automated remediation

5

Why build Beringei?

• Size requirements in 2013 when we started building Gorilla• 2 billion unique time series

• Real-time analysis prioritizes most recent day of data: 16TB

• With lossless compression 1.3TB

• In 2015 we presented Gorilla at VLDB• 4 billion unique time series

• Most recent day of was about 32TB uncompressed, 2.6TB compressed

6

Why build Beringei?

• Beringei is the open-source Gorilla• Big effort to abstract the “Facebook specific” infra out of Gorilla

• Few external dependencies, all of which are open source

• Today, Beringei is used in the monitoring system at Facebook• 10 billion+ unique time series

• 18 million queries per minute

System Overview

8

System Overview: Compression

• Values and timestamps are compressed separately

• Efficient and simple streaming compression

9

System Overview: CompressionValue compression, cont

Valu

e

Control Bits Leading 0s Data Length Data

12 0 0001 00 1100 1000 0000 0101

24 11 0 1011 00 0001 1

15 11 0 1011 00 0100 1011

12 10 0011

12 0

10

System Overview: Compression

11

System Overview: CompressionTime Delta Delta of

Delta

Control Bits Data

1441000000 0101 0101 1110 0011

1110 1010 0100 0000

1441000060 60 00 0000 0011 1100

1441000120 60 0 0

1441000181 61 1 10 000 0001

1441000240 59 -2 10 111 1110

12

System Overview: Compression

13

Compression Results

14

Compression Results

16TB -> 1.3TB

Comparison with PrometheusComparing with a well-known, open source TSDB and monitoring platform

16

Beringei vs. Prometheus

Prometheus feature set

• High-dimension data model

• Flexible data model

• Efficient on-disk/in-memory

storage model

• Built-in data collection

• Support for Alertmanager

Beringei feature set

• Key-value store (for time series)

• In-memory only

17

Beringei vs. Prometheus

So … who should use Beringei at all?

• If your existing monitoring system is no longer fast enough

• If you are a developer for Prometheus (or other monitoring system) and want to accelerate your system

18

Beringei

Beringei is super fast! A single server can:

• Serve up to 3k queries per second• 95th percentile read latency of 65us

• Sink up to 1.5M points per second• Streaming compression achieves ~90% compression at 1.5M pps

• Average 300us write-to-read availability latency

• Hold > 100M unique time series• Averaging more than 1M time series stored per GB of ram

How to use Beringei

20

How to use Beringei

Get involved!

https://github.com/facebookincubator/beringei

Can be used as a library within your existing project

Includes a reference service implementation

Included thrift-based server and clients

Simple http service that directly supports Grafana

21

Thank you!

Questions?