Database story by DevOps

A Database story by DevOpsexploring SQL Database in Production.

Anton Martynenko

Dev & QA & OPS = DevOpsHead of Technical Operations at Youscan

.Net, Windows Azure, Sql Server, NoSql…

Twitter: @aamartynenkoam@youscan.biz

YouScan is a leader of the emerging social media monitoring & analytics market in Russia and Ukraine.

Our products:http://youscan.ruhttp://leadscanner.ru

The Reality:

Time is limited

Talk covers a lot of stuff

No time to go into details

If you have a question – make notes and ask later

The Production

Main DB

WWWAndroid

appService iOs app

The Production with more databases

Main DBClientsTools

WWWAndroid

appService iOs app

The Production

Main DB

WWWAndroid

appService iOs app

Database as a Black Box

WWWAndroid

appService iOs app

Database as a source of problems

Main DB

WWWAndroid

appService iOs app

Main DB

Firewall for production DB

WWWAndroid

appService iOs app

DevOps – Developer with access to Prod

Main DB

WWWAndroid

appService iOs app

Typical Database headaches

1.Performance

2.Whatever-SQL mess in DB

3.Maintenance

Main DB

Typical Database headaches

1.Performance

2.Whatever-SQL mess in DB

3.Maintenance

Main DB

Performance issues

Memory

Disk I/O bottlenecks

High CPU usage

Know your Production hardware!

- Memory

- Disk I/O

- CPU usage

Understand your load

Latencies are important

Understand Database “Math”

Tables

Indexes

Queries

Understand Transaction Isolation

Serializable

Repeatable reads

Read committed

Read uncommitted

Performance: User Experience

WWW

“Place order”

“Success”

Main DB

User’s Transaction is complex

WWW

“Place order”

“Success”

Main DB

User’s Transaction is complex

WWW

“Place order”

“Success”

Main DB

So Slow!

Performance: TOP I/O Queries

Performance: TOP CPU Queries

Google to find TOP IO/CPU Queries

Catch the bottleneck

CPU Score Query

13543541 SELECT * FROM Products WHERE …

7545314 SELECT * FROM Users where …

567541 INSERT INTO Reviews …

IO Score Query

68743513 SELECT * FROM Authors INNER JOIN...

1475214 INSERT INTO OrderQueue …

1246876 INSERT INTO SystemLogs …

Analyze Top IO Queries

Analyze Top CPU Queries

Understand your query

Understand indexes used by query

Understand execution plan

Slow “SELECT”

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Slow “INSERT”, “UPDATE”

Too many Indexes

Suboptimal Query

Low disk IOPS (or too many inserts)

Understand the query

Catch!

Where is the problem?

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Too many Indexes

Root cause: missing index

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Too many Indexes

Review existing indexes Add new index Or change existing

index

Root cause: query

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Too many Indexes

Understand execution plan Use Sql query parameters Do select/update minimal

field set

Root cause: low memory

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Too many Indexes

Review fields in index Archive old data Check DB Engine config Add more memory

Root cause: low IOPS

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Too many Indexes

Use partitioning Go to SSD Get rid of queries

Root cause: too many indexes

Missing Index

Suboptimal Query

Low Memory (or Big Index)

Low disk IOPS (or too many queries)

Too many Indexes Remove unused Remove features

Many indexes = many features

Remove uselessfeatures

Analyze user behavior

Big Table

Table Partitioning

Jan 2014

Feb 2014

Mar 2014

Apr 2014

May 2014

June 2014

Table Orders, All Data

Partitioned table Orders, Partition = Month

Database Sharding Multiple instances,Shard per country

Main DB

GB Main DB

DE Main DB

FR Main DB

Single instance,All Data

Multitenancy Multiple instances,Shard per client

Main DB

Coca Cola Main DB

BMWMain DB

ShellMain DB

Single instance,All Data

NestleMain DB

Macdonald’sMain DB

SiemensMain DB

DB License very expensive

Coca Cola Main DB

BMWMain DB

ShellMain DB

NestleMain DB

Macdonald’sMain DB

SiemensMain DB

Typical Database headaches

1.Performance

2.Whatever-SQL mess in DB

3.Maintenance

Main DB

Databases do a few things really well:

• They store and retrieve data

• They enforce relationships between data entities

• They provide the means to query the data for answers

T-SQL, PL-SQL, Whatever-SQL are poor as programming languages

SQL programming tools sux

Debugging and testing is not easy…

Typical T-SQL or PL-SQL mess in Database

What about Refucktoring your SQL code?

Typical T-SQL or PL-SQL mess in Database

Java/C#/etc. with modern tools

Typical T-SQL or PL-SQL mess in Database

WTF your code is doing in Database?

There are no reasons to put your Business Logic

in Database

Data -> Database

Data access -> DAL

Transaction -> (not sql)Code

Deadlocks

Complex code --> Deadlocks

ORM

ORM

Main DB

Application

In Legacy systems more often than notORM adds complexity to the system

ORM

ORM is magic

?

Main DB

Application

Main DB

Lightweight ORM

Application

Typical Database headaches

1.Performance

2.Whatever-SQL mess in DB

3.Maintenance

Main DB

Manage schema changes

Change = Script

Two-staged schema changes

Antipattern: Configuration in DB

WWWAndroid

appService iOs app

Antipattern: Lookup tables in DB

Antipattern: Queuing in Database

Keep Schema clean and simple

Get rid of configuration in DB

Get rid of lookup tables in DB

Get rid of write contention

Long term goals:

Database size

Restore/backup

10 minutes

1 Hour

12 Hours

X 3 days

Do Archive old data

Trust is foundation for everything

Thanks! Questions?

Me: https://www.linkedin.com/pub/anton-martynenko/10/289/9b5

YouScan - http://youscan.ru/LeadScanner - http://leadscanner.ru/

Monitoring & Metrics:Zabbix - http://www.zabbix.com/KissMetrics - https://www.kissmetrics.com/Kibana - http://www.elasticsearch.org/overview/kibana/