Upload
bob-burgess
View
198
Download
0
Embed Size (px)
Citation preview
Storage Methodsfor
Nonstandard Data PatternsB O B B U R G E S S
D ATA B A S E O P E R AT I O N S E N G I N E E R
2
AgendaIntro
Choices Available To You
Choices I Tested
Testing Methods
What I Saw When Loading Data
What I Saw When Using Data
Storage Engine Comparison
Conclusions
3
W H O I S T H I S G U Y ?
4
Bob Burgess
Developer, then DBA
1997, Oracle & Sybase
2007, MySQL at Radian6 => Salesforce
2015, MySQL at Shopify
2006 Platform to let merchants easily sell online and in person 243,000+ Active stores / $14B in sales / 150 Countries Large MySQL Installation Hiring!
5
6
Dont Panic
All slides will be posted to the conference site & Slideshare
Test scripts and test data will be posted to Github bobburgess
7
Why Am I Doing This?
I cant be the only one
Surprising results
8
C H O I C E S
9
Normal Tables
A few numbers and average-size varchar columns
Couple of indexes
Artificial or random primary key
10
Unusual Tables
Very tiny rows
Very large rows
Large random primary key
Unusual Tables Very tiny rows
Very large rows
Large random primary key
11
Multiple TB
Two bigints
Generally-increasing primary key
InnoDB
Unusual Tables Very tiny rows
Very large rows
Large random primary key
12
De-duplication
SHA1 hash
binary(20)
Completely random primary key
InnoDB
Unusual Tables Very tiny rows Very large rows
Large random primary key
13
De-duplication
SHA1 hash
binary(20)
Completely random primary key
InnoDB
Yves Trudeau (Percona)
Auto-increment PK
Index on 1st 5 bytes of binary(20) hash
Trigger to check for Duplicates
14
Storage Engines
MySQL
client
disk
15
Storage Engines
MySQL
client
disk
MySQL
client
disk
Storage Engine
16
Storage Engines
MySQL
client
disk
MySQL
client
disk
Storage Engine
17
Storage Engines
MySQL
client
disk
MySQL
client
disk
Storage Engine
18
Choices Storage: Solid-state vs. Spinning Rust
Arrays of Storage (RAID levels)
Storage Engine
suitability for data shape
suitability for access patterns
adaptability to slower storage
19
MY T E S T SC H O I C E S T E S T E D ( VA R I A B L E S )
20
Primary Key
Auto-Increment (Sequential) Integer
Random Integer
Random Large (20 bytes)
Auto-Increment (Sequential) Integer With Trigger
21
Data Shape
22
Data Shape: Tiny Rows
create table tinyrows ( id bigint primary key, payload bigint);
23
Data Shape: Tiny Rows
create table tinyrows ( id bigint auto_increment primary key, payload bigint);
24
Data Shape: Tiny Rows
create table tinyrows ( id bigint auto_increment primary key, payload bigint, index i (payload));
25
Data Shape: Average Rowscreate table averagerows ( id bigint primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50));
26
Data Shape: Average Rowscreate table averagerows ( id bigint auto_increment primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50));
27
Data Shape: Average Rowscreate table averagerows ( id bigint auto_increment primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50), index i1 (num0), index i2 (char1), index i3 (char3), index i4 (num2));
28
Data Shape: Average Rowscreate table averagerows ( id binary(20) primary key, num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50), index i1 (num0), index i2 (char1), index i3 (char3), index i4 (num2));
29
Data Shape: Average Rowscreate table averagerows ( newid bigint auto_increment primary key, real_id binary(20), num0 bigint, char1 varchar(60), num1 decimal (10,2), num2 double, char2 varchar(60), char3 varchar(50), index id_helper (real_id(6)), index i1 (num0), index i2 (char1), index i3 (char3), index i4 (num2));
30
Data Shape: Average Rows
create trigger averagerows_bi before insert on averagerows for each row begin select count(*) into @c from averagerows where real_id = new.real_id; if @c != 0 then set @m=concat('Duplicate entry ''',hex(new.real_id),''' for key ''PRIMARY'''); signal SQLSTATE '23000' set message_text=@m, mysql_errno=1062; end if; end;
31
Data Shape: Average Rows
create trigger averagerows_bi before insert on averagerows for each row begin select count(*) into @c from averagerows where real_id = new.real_id; if @c != 0 then set @m=concat('Duplicate entry ''',hex(new.real_id),''' for key ''PRIMARY'''); signal SQLSTATE '23000' set message_text=@m, mysql_errno=1062; end if; end;
Do we have this one already?YES
Raise an error
32
Data Shape: Large Rows
create table largerows_ai ( id bigint primary key, payload1 varchar(2000), payload2 varchar(2000), payload3 varchar(2000), payload4 varchar(2000));
33
Data Shape: Large Rows
create table largerows_ai ( id bigint auto_increment primary key, payload1 varchar(2000), payload2 varchar(2000), payload3 varchar(2000), payload4 varchar(2000));
34
Data Shape: Large Rows
create table largerows_ai ( id bigint auto_increment primary key, payload1 varchar(2000), payload2 varchar(2000), payload3 varchar(2000), payload4 varchar(2000), index i1 (payload1(40)), index i2 (payload2(767));
35
Storage Engines
36
Storage Engine: InnoDB
Very Mature
Well understood
Continually updated
Supported by Galera Cluster
37
Storage Engine: TokuDB Was separately-licensed; bought by Percona last year
Built into Percona Server now
Extreme data compression
Online table alters
Other benefits
Not supported by Galera Cluster
38
Storage Engine: Deep Separately-licensed product
Distributed as a complete package, based on 5.6
Does data compression
Self-tuning using constant data analysis
Other benefits
Not supported by Galera Cluster
39
Filesystem
ext4, xfs, others
Journaling uses IO and can be disabled
40
T E S T I N G M E T H O D S
41
HardwareLoad Generator Database
1 Gbe
Dell 2950-III 32 GB RAM Two 4-core 3GHz Xeon processors SSD and HDD storage Ubuntu 14.04 Measured SSD IOPS (fio: random RW / 5G file): 9750 read, 3254 write
Desktop 8 GB RAM Intel i7 processor SSD storage Ubuntu 14.04
42
SoftwareLoad Generator Database
Ubuntu 14.04 Percona Server 5.6.28 deepSQL 5.6.28-21214
Ubuntu 14.04 go v.1.2.1
43
Data Loading in go for speed
Loop until stopped: Fill buffer with 1000 (or 10000) rows Send buffer to LOAD DATA INFILE
Very low CPU and network on load-gen box
Concurrency of 1 and 20
github.com/bobburgess
http://github.com/bobburgess
44
Data Usage (Work that DB) in go for speed
Loop until stopped: Select by PK Select by each secondary index
(column starts with random value) Insert a row of random data
Very low CPU and network on load-gen box
Concurrency of 20, 60, and 100
github.com/bobburgess
http://github.com/bobburgess
45
DB Server Measurements
in bash cause Im lazy (and speed not required)
Loop until stopped: Take measurements Output measurements Remember old values (for differential measurements) sleep for 1 sec minus the time taken for the measurements
Very lightweight
github.com/bobburgess
46
DB Server Measurements
Rows per second inserted & selected
Disk IOs per second, read & write
Bytes per second, read & write
Size of mysqld process
Size of data on disk
CPU %, User-time & System-time
47
DB Server Measurements
Rows per second inserted & selected
Disk IOs per second, read & write
Bytes per second, read & write
Size of mysqld process
Size of data on disk
CPU %, User-time & System-time
Measure of throughput
Possible to use slower storage?
Possible to use slower storage?
Wasteful of memory? Bloated?
Possible to use less storage?
Lower = can handle more load
48
Testing Flow
Start DB monitor on DB server (output to file)
Start up required processes on load-gen box
Check DB server for unexpected activity
Check load-gen box for unexpected activity
Check load-gen box for CPU or network bottlenecks
Stop test processes when data set large enough
49
Plotting the Data
R
Define columns columns
50
Plotting the Data Example: Two plots with running-average lines
par(mar=c(5, 9, 4, 6) + 0.1)options(scipen=15)Xmin=1Xmax=14432Ymin=0Ymax=120smooth=100# Firstplot(seq(Xmax), avg_ai_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], col="red", pch=".", xlab="", ylab="", ylim=c(Ymin,Ymax), type="p", axes=FALSE, main="")axis(2,ylim=c(Ymin,Ymax), col="red", las=1)mtext("IOPS - Write", side=2, line=3)box()lines(seq(Xmax), filter(avg_ai_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], rep(1/(smooth+1),(smooth+1)), sides=2), col="red", lwd=2)par(new=TRUE)# Secondplot(seq(Xmax), avg_rnd_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], col="blue", pch=".", xlab="", ylab="", ylim=c(Ymin,Ymax), type="p", axes=FALSE)lines(seq(Xmax), filter(avg_rnd_i_load_20ses_ssd_inno$IOPSW[Xmin:Xmax], rep(1/(smooth+1),(smooth+1)), sides=2), col="blue", lwd=2)# Finish upaxis(1,at=seq(0,Xmax,60),labels=seq(0,Xmax/60))mtext("Minutes", side=1, col="black", line=2)legend("topright",c("Auto-Inc PK","Random PK"),fill=c("red","blue"))
51
R E S U LT S
InnoDB
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
56
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
57
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
58
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
59
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
60
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
61
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
62
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
200,000
63
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
64
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
65
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
66
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
67
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
68
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
69
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
70
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
71
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
72
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
73
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
82
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
83
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
84
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
85
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
?!
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
93
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
94
95
InnoDBBulk-Load
Single-Session 1 Multi-Session 0
Primary Key
Auto-Inc 1 Random 1 Large 0
TokuDB
96
97
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
98
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
99
100
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
101
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
102
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
103
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
40-60
104
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
105
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
106
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
107
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
108
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
109
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
110
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Difference with InnoDB is not as great
111
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
112
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
113
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
114
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
115
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
116
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
117
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
118
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
119
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
120
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
121
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
122
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
123
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
124
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
125
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
126
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
127
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
128
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
129
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
130
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
131
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
132
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
133
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
134
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
135
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
136
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
137
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
138
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
139
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Same shape as PK-Only, 1 Session
146
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
147
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
148
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
deepSQL
153
154
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
InnoDB
TokuDB
157
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
158
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
161
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
162
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
163
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
164
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
165
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
166
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
167
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
168
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
171
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
172
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
173
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
174
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
175
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
176
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
177
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
178
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
179
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
180
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
181
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
182
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
183
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
184
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
185
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
186
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
InnoDB
TokuDB
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
194
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
195
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
196
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
197
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
198
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
199
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
200
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
201
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
202
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
203
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
204
Storage Engine Row Size________ Primary Key_________ Indexing Sessions InnoDB TokuDB Tiny Average Large Auto-Inc Auto-Inc+Trig PK Only 1 20 Deep Random Random Lrg Multiple
205
Usage Performance
206
Usage Performance Each row size
Auto-Increment PK
Inserted by multiple threads
disk files of 2X or 3X cache size
207
Usage Performance
Loop this block: Several selects One Insert
20, 60, 100, 200 concurrent
Count Com_insert and Com_select on server side
208
Usage Performance
Average Rows InnoDB TokuDB Deep
Rows 3,264,053 6,742,470 5,324,183
Avg. Row Length 366 174 180
~2 GB on disk
209
Tiny Rows (Select)
20 sessions 60 sessions 100 sessions 200 sessions
210
Tiny Rows (Insert)
211
Average Rows (Select)
212
Average Rows (Insert)
213
Large Rows (Select)
214
Large Rows (Insert)
215
N O M O R E C H A R T S !
216
What I Learned
Vast differences depending on use-case
Some patterns not as bad as first thought (i.e. Random PK)
Some patterns worse than first thought (i.e. with bytes/row)
217
Take-Aways
Test for your use-case
Dont assume you already know how the DB will respond
Dont trust anyone, especially me. Test it yourself!
218
T H A N K Y O U !S L I D E S W I L L B E O N L I N E
B O B . B U R G E S S @ S H O P I F Y. C O M G I T H U B . C O M / B O B B U R G E S S
P L E A S E F I L L O U T A N E VA L U AT I O N !
is hiring!