Scaling DB Access for 100s of billions of queries per day

TLDR - Hera

Open Source

High Eff iciency

Reliable Access

to data stores

Out Of DB Connections?

• Scaling

• Multiplexing

• Sharding

• Resiliency

• Manageability

• Our switch to Go Lang

About Petrica

• Works @ PayPal

• developing the database proxy.

• database backed publish-subscriber messaging system

• PayPal Here backend

• Golang contributor

• Previously @

• Worked on distributed systems, embedded system, desktop apps

About Kenneth

• Works @ PayPal

• paypal.de

• PayPal Verisign Two Factor Authentication

• Coded shared memory for sharding

• Migrated to Go (hera)

• Previously @

• Visualization of Genomic Metabolism, Artificial Chemistry, and EPICS

Monitoring

About PayPal – 2018 facts

100+Currencies

200+Markets

267MActive Customer Accounts

9.9BPayments Transactions

$578B Total Payments Volume

Up 27% YoY*Total Payment Volume

2,700Applications

4,500Engineers

17,000Releases

PayPal OLTP Environment scale

2000+ Database Instances

100+ Billion Calls/day

3000+Database Nodes

74 PB Total Storage

300+ Schemas

10 +Availability Zones

200,000 +Application Servers

1+ Billion Calls/hour

Key Take-aways

• Massive scale

• Complexity

• Growth

Mic ro se rv i ce

connect

request

disconnect

request

response

Typical Database Access by Application

Mic ro se rv i ce

connect

request

disconnect

request

response

100 ms

103 ms

Mic ro se rv i ce

connect

request

response

100 ms

Po o l

Connection is pre-established, cached and re-used for each request

release

From Few to Thousands Microservices

Mic ro se rv i ce

Configure the pool cache large

enough for availability

Mic ro se rv i ce

M i c ro se rv i ce

Add more service when scaling up

Mic ro se rv i ce

M i c ro se rv i ce

Mic ro se rv i ce

M i c ro se rv i ce

Mic ro se rv i ce

M i c ro se rv i ce

Hit a connection limit

Solutions?

• Lower the connection pool

size in each microservice

• Improve microservice code

to use the connection more

efficiently

• Buy better database

hardware

Partitioned Connection Pool

Mic ro se rv i ce

M i c ro se rv i ce

Request

While servicing requests, the

microservices use only some

DB connections

Mic ro se rv i ce

M i c ro se rv i ce

Request

DB connections

In some node all the

connections can be temporary

Mic ro se rv i ce

DB connections

In some node all the

connections can be temporary

Client requests will incur

latency though the database

has enough capacity

Mic ro se rv i ce

Request

Solutions?

Scale with shared pool !!!

S h a r e d c o n n e c t i o n p o o l

Mic ro se rv i ce

M i c ro se rv i ce

Request

Shared pool

Se rv i ce

Service request

Service

response

SQL request

DB connection

Database connection is

allocated but mostly idle

Typical service request

Se rv i ce

Service request

Service

response

SQL request

DB connection

Serving other requests when client seems idle?

Multiplexing

Mic ro se rv i ce

M i c ro se rv i ce

Request

RequestSha red Po o l

Services perceive that they have a database connection

Multiplexing

Mic ro se rv i ce

M i c ro se rv i ce

Request

RequestSha red Po o l

Services perceive that they have a database connection

Hera – Highly Efficient Reliable Access

Mic ro se rv i ce

M i c ro se rv i ce

Request

Shared connection pool multiplexer

to data s tores

HERAMicroservice-

DB proxy

Principles and constraints

• Correctness

• Resiliency, -ilities, performance

• Minimal configuration

• Thin client

• Minimal change to wire protocol

Let’s talk about other features!

Features

• Multiplexing

• Read write split

• Sharding

Read-Write Split

Oracle RAC - Real

Application Cluster

Each database is exchanging

lock information

Read-Write Split

Oracle RAC - Real

Application Cluster

Hera No de

Separate write queries

and direct to one node

Reduces traffic between

DB nodes

Sharding

• Client vs Server

• Finding the Shard

• Sample Query Run

• Convert Legacy App

• Move Scuttle Bin’s Data

Sharding: Client vs Server

C++ Client

C++ Lib

Database (shard 0) Database (shard 1) Database (shard 2)

Sharding: Same Logic in Java Client

C++ Client

C++ Lib

Java Client

Java Lib

Sharding: Duplicate Logic in Go Client

C++ Client

C++ Lib

Java Client Go Client

Java Lib Go Lib

Sharding: Copy Logic in Python

C++ Client

C++ Lib

Java Lib Go Lib

Python Client

Python Lib

Sharding: Node.js Client Too

Node.js ClientC++ Client

C++ Lib

Database (shard 1) Database (shard 2)

Java Lib Go Lib

Python Client

Python Lib Node.js Lib

Database (shard 0)

WorkerWorker

Node.js Client

Sharding: Keep Logic in Server

Hera Multiplexer

C++ Client

C++ Lib

Hera Java Lib Hera Go Lib

Python Client

Python Lib Node.js Lib

Sharding: Implementation

1. Shard Key: account_id=2000

Sha rd 0 Sha rd 1 Sha rd M

Hash function (shard key value) %K

Scuttle bin to logical shard mapping

Logical to physical shard mapping

Shard Key

DB Shard

Shard Key

DB Shard

2. Murmur3Hash(2000)%1024=280

Shard Key

DB Shard

2. Murmur3Hash(2000)%1024=280

3. Select shard_id from

hera_shard_map where

scuttle_id = 280;

shard_id

---------

Shard Key

DB Shard

2. Murmur3Hash(2000)%1024=280

3. Select shard_id from

hera_shard_map where scuttle_id =

shard_id

---------

4. TWO_TASK_1 = LOAN_SH1

TWO_TASK_1=‘tcp(loan-sh1:

3306)/loan’

Sharding: Converting Application Query

- Shard Key: account_id

- select * from loan, appfile where loan.id = ? and appfile.loan_id = loan.id

Sharding Compatible Query

- select * from loan, appfile where loan.id = ? and appfile.loan_id = loan.id and loan.account_id = ? and appfile.account_id = loan.account_id

Sharding: Hera JDBC SQL Rewrite

- select * from loan, appfile where loan.id = :loan_id and appfile.loan_id = loan.id and loan.account_id = :account_id and appfile.account_id = loan.account_id

- Hera can now pick out the shard key name and value

Sharding: Legacy App Conversion

• All queries directed to shard 0

• Logs queries that don’t bind to shard

Sha rd 0

Shard Key

DB Shard

Sharding: Legacy App Conversion

• Whitelist sends one value to a

specific shard

• Limits risk of failures to 1 value

• Hera uses Shard 1, but same DB

• Fast rollback on errors

• Repeat for larger sets

Sha rd 0

Hash function (shard key value)

Scuttle bin to logical shard

mapping

DB Shard

Whitelist

Sha rd 1

Sharding: Physical Whitelist

• Validates permissions and data

Sha rd 0

mapping

DB Shard

Whitelist

Sha rd 1

Sharding: Logical Move for One Scuttle Bin

• If successful, do a physical data move next

Sha rd 0

mapping

DB Shard

Whitelist

Sha rd 1

Sharding: Moving Scuttle Bin

• Start data copy

• Typically, tables are partitioned by

scuttle bin

Shard Key

DB Shard

• Start data copy

scuttle bin

• Block writes

Shard Key

DB Shard

• Start data copy

scuttle bin

• Block writes

• Data fully copied

Shard Key

DB Shard

• Start data copy

scuttle bin

• Block writes

• Data fully copied

• Update map

• Use scuttle bin in new location

Shard Key

DB Shard

Sharding: Take Away

• Queries must bind shard key & value

• Manage risk converting legacy code

• Controlled data redistribution

Resiliency

Principles

• Deal with exceptional

▪ Surge in load

▪ Slow SQL requests

▪ Issues with the DB

Resiliency

Principles

• Deal with exceptional

▪ Surge in load

▪ Slow SQL requests

▪ Issues with the DB

• Limit or avoid impact

• Self heal

Resiliency

Features

• Bouncer

• Surge Queue

• Slow Query Eviction

• Read Replica

• Transparent Application

Failover

Resiliency: Oversubscription - Bouncer

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Se rv i ce 4

Service 4 connection is

tcp accepted and then

closed without SSL.

Resiliency: Oversubscription – Surge Queue

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Service 3 can make a

request

It waits--queued

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Generally, 1ms later,

Service 3 gets run.

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Waiting more than 1s gives

an error to Service 3.

Resiliency: Slow Query Eviction

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Slow Query

When Hera is overloaded

Resiliency: Slow Query Eviction

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Slow Query

Hundred of queries can run

once slow query is

removed.

Monitoring Usage

Se rv i ce 1

Hera No de

Se rv i ce 2

Se rv i ce 3

Each Hera NodeFree DB

Connections

(acpt)

Resiliency: Eviction – Actual event

Each Hera NodeFree DB

Connections

(acpt)

Happened few years back, when automatic eviction went unnoticed.

Evicting expensive query makes space for many typical queries.

Read Replicas

MUX R1 Pool

Po o l 1

R1 PoolR1 Pool

MUX R1 PoolR1 Pool

R2 Pool

Po o l 2

Appl icat ion

Read Repl ica 1

Read Repl ica 2

Wri te DB

F a i l o v e r

m o d u l e

B u s i n e s s

l o g i c

R1 & R2: different data replication lag

Transparent Application Failover – Read Replica Retry

R1 PoolR1 Pool

R1 Pool

R1 PoolR1 Pool

R2 Pool

Po o l

Appl icat ion

B u s i n e s s

l o g i c

Read Repl ica 1

Read Repl ica 2

Write DB

• Build failover on server side

• Also failover when query is slow

Manageability

• Maintenance without customer

impact

• Recommend clients recycle

• Oracle RAC maintenance

Recycle Connections– Node Going Away

Hera No de

Se rv i ce 2

Se rv i ce 3

Hera No de

TCP Load

Balancer

Recycle Connections for Manageability

Hera No de

Se rv i ce 2

Se rv i ce 3

Hera No de

TCP Load

Balancer

Recycle Connections– Node Returns

Hera No de

Se rv i ce 2

Se rv i ce 3

Hera No de

TCP Load

Balancer

Hera No de

DB Maintenance

Oracle RAC - Real

Application Cluster

Hera No de

• Preparing Oracle RAC node 3

maintenance

• DBAs remove node 3 from Oracle

configs

• Insert into hera_maint (inst_id,

status, status_time, module,

machine) values (3, ‘F’, [unix

epoch]+2, [hera pool name], [host]

DB Maintenance

Oracle RAC -Real

Application Cluster

Hera No de

• After [unix epoch] +2

Take Aways

• Keep controls near those who need

• Avoid being unnecessarily involved

Transition to Go

- Written in C++, asynchronous code

- Scalable

- Efficient

- Complex

Previous version

Transition to Go

- Scalable

- Efficient

- Complex

Previous version

Transition to GoNew changes

- New functional requirements

- New non functional requirements (efficiency)

- Scalable

- Efficient

- Complex

Previous version

Transition to GoNew changes

- New functional requirements

- New non functional requirements (efficiency)

- Go programing is synchronous

- Scale efficiently out of the box

- Does it meet our requirements?

- Is stop-the-world garbage collection time an issue?

How about writing in Go?

Proof-of-concept

1st Release: No Sharding

Full feature with C++

dependencies

Full feature all Go

Going with Go

Transition to Go

- Running in production for 1 year

- Latency and CPU met the requirements

- No clients change

- Had some issues and learnings

Mission Accomplished

func routine() {

time.Sleep(time.Second * 10)

func main() {

routines := 10000

wg.Add(routines)

for i := 0; i < routines; i++ {

go func() {

routine(i)

wg.Done()

wg.Wait()

Spawn 10000 go-routines sleep for 10 seconds to simulate processing

Running on my 2 cores test machine

How many OS threads I expect at runtime?

Learning Example

func routine() {

time.Sleep(time.Second * 10)

func main() {

routines := 10000

wg.Add(routines)

go func() {

routine(i)

wg.Done()

wg.Wait()

Spawn 10000 go-routines sleep for 10 seconds to simulate processing

Learning Example

$> ./mytest &

[1] 12345

$> ps –L –C mytest –no-headers | wc –l

func routine() {

//time.Sleep(time.Second * 10)

syscall.Select(0, nil, nil, nil, &syscall.Timeval{Sec: 10})

func main() {

routines := 10000

wg.Add(routines)

go func() {

routine(i)

wg.Done()

wg.Wait()

Learning ExampleSpawn 10000 go-routines sleep for

10 seconds to simulate processing

Using system call to sleep how many OS threads I expect at runtime?

func routine() {

//time.Sleep(time.Second * 10)

syscall.Select(0, nil, nil, nil, &syscall.Timeval{Sec: 10})

func main() {

routines := 10000

wg.Add(routines)

go func() {

routine(i)

wg.Done()

wg.Wait()

Learning ExampleSpawn 10000 go-routines sleep for

10 seconds to simulate processing

$ ./mytest

runtime: program exceeds 10000-thread limit

fatal error: thread exhaustion

runtime stack:

. . . “A Go program creates a new thread only when a goroutine is

ready to run but all the existing threads are blocked in system calls,

cgo calls, or are locked to other goroutines due to use of

runtime.LockOSThread”

from “runtime/debug” func SetMaxThreads()

Learning Example

Key Take Away • System calls lock the OS

thread. Ex: os.File read

• When using C/C++ library,

be aware of the systems

calls (mutexes, file reads,

socket reads, sleep)

• Golang locks, socket reads,

sleep are not locking the OS

thread

• In your load tests, verify the

number of OS threads

Open source

github.com/paypal/hera

✓ Code, documentation, examples

✓ Standard clients

• Java: JDBC driver

• Golang: database/sql driver

Google group heradatalink

github.com/paypal/hera

Internal Mirror

Deploy

PayPal Config

Acknowledgements

Kamlakar

Mukundan

Yaping

Shuping

Stephanie

Paresh

Dheeraj

Pradeep

Pramod

Samrat

Ashwin

and countless others

Next Steps

Need more database connections?

Hera can help scale

• Multiplexing

• Sharding

• Resiliency

• Manageability

Open Source github.com/paypal/hera

Google Groups – heradatalink

Our Future

• Scaling Tools

• More Clients

github.com/paypal/hera Google group: heradatalink

Scaling DB Access for 100s of billions of queries per day · 25.06.2019 · Microservice connect...

Documents

WAIS Mainframe emergency access request process

Access Request Process - my.education.nsw.gov.au › help › platoapps › media › documents › ern › ... · Access Request Process May 2016User Guide 4 / 211 Disability, Learning

ISIS Access Request

Data Access Request Tracker (DART)

G DIN XI S RАЈЕVО, 25.06.2019. BRОЈ / NUMBER 5bhas.gov.ba/data/Publikacije/Saopstenja/2019/IND... · 1 3 gОdinА / year xi sАrАЈЕvО, 25.06.2019. brОЈ / number 5 pОslОvnЕ

Request for Proposals – Data Governance and Access 11 Data Governance and Ac… · Request for Proposals – Data Governance and Access . 2017RFP-11 Data Governance and Access

Request for Access to Criminal History Record Information › ojis › documents › ... · Request for Access to Criminal History Record Information . Access to criminal history

Volcano Mesa Access Modification Request

Peoplesoft (ERP) Campus Solutions Access Request Form …...PeopleSoft ERP Campus Solutions Access Request Form User Guide FINAL Last Revised: 08/08/13 Page 4 of 21 1.0 Access Request

A Public Records Access request has been submitted....PRA #19007 [P1 A Public Records Access request has been submitted. Request By: Ryan Franco, Esq. Signature: Ryan L. Franco Request

Freedom of Information Access Request - Toronto · 2019. 6. 13. · Freedom of Information Access Request Use this form to: Request City information Request personal information or

Making an access-to-information request

ELECTRONIC BUILDING ACCESS REQUEST FORM

How to request Online Access - Royal Cup Coffee · HOW TO REQUEST ONLINE ACCESS A walkthrough on how an existing customer can request online access

Request TEASE Access Steps to request access to TREx (Texas Records Exchange) though TEASE (Texas Education Agency Secure Environment)

Subject Access Request Process V1 - Patients & · PDF fileNHS Borders Information Governance Subject Access Request Process V1 .doc Page 3 of 37 Subject Access Request Process Summary

LAW ENFORCEMENT DATABASE ACCESS REQUEST FORM

#8. Appendix Request Access to Systems

Subject Access Request Process V1 - NHS Borders

A Public Records Access request has been submitted.corpinfo.panynj.gov/files/uploads/documents/freedom-of...1 A Public Records Access request has been submitted. Request By: Edward