47

MySQL For Linux Sysadmins

Embed Size (px)

DESCRIPTION

MySQL

Citation preview

Page 1: MySQL For Linux Sysadmins
Page 2: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

MySQL  For  Linux  Sysadmins

Morgan  Tocker  MySQL  Community  Manager  August,  2014

Page 3: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Safe  Harbor  Statement

The  following  is  intended  to  outline  our  general  product  direction.  It  is  intended  for  information  purposes  only,  and  may  not  be  incorporated  into  any  contract.  It  is  not  a  commitment  to  deliver  any  material,  code,  or  functionality,  and  should  not  be  relied  upon  in  making  purchasing  decisions.  The  development,  release,  and  timing  of  any  features  or  functionality  described  for  Oracle’s  products  remains  at  the  sole  discretion  of  Oracle.

Oracle  Confidential  –  Internal/Restricted/Highly  Restricted3

Page 4: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted4

Program  Agenda

Introduction  

MySQL  Architecture  

Backup  &  Replication  

Performance  &  Capacity  Planning

1

2

3

4

Page 5: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

The  Career  Path  to  DBA

• Start  as  a  Developer  • Start  as  a  Sysadmin  • Train  to  be  a  DBA

5

Usually either #1 or #2. I started as a Developer}

Page 6: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Non  Exhaustive  List  of  Duties

• Backup  • Replication  • Patching  &  Upgrades  • Performance  and  Capacity  Planning  • Coaching  &  Mentoring  • Fighting  Fires

6

Page 7: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted7

Program  Agenda

Introduction  

MySQL  Architecture  

Backup  &  Replication  

Performance  &  Capacity  Planning

1

2

3

4

2

1

Page 8: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

About  MySQL

• Version  numbers  are  misleading:  • 5.5  to  5.6  was  the  biggest  upgrade  in  history  • 5.6.xx  

• Micro-­‐release  approximately  every  2  months.  • Fixes  bugs  and  security  vulnerabilities.

8

Page 9: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

About  MySQL  (cont.)

• 5.7.xx  • Current  focus  of  development.  • Release  cycle  is  2-­‐3  years.  

• 5.5  and  5.6  are  under  active  support.

9

Page 10: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted10

Page 11: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Storage  Engines  in  MySQL

• InnoDB  (default)  • MyISAM  (previous  default)  • NDB  Cluster  • Other

11

Page 12: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Memory  Usage

• Databases  like  to  allocate  large  amounts  of  memory  for  caches  

• Better  to  run  with  smaller  caches  than  swap  • Kind  of  works  as  “double  buffering”  of  operating  system  caches

12

Page 13: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

MySQL  Server

• Single  process  of  mysqld  • Supervisor  process  is  mysqld_safe  • Each  new  MySQL  connection  consumes  a  thread  

• With  the  server  maintaining  a  small  thread  cache

13

Page 14: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted14

Transactio

nSystem

Storage

Caching

SYS_TABLES

ibda

ta1

spac

e 0

Page Cache

A.ibd

B.ibd

C.ibd

IBUF_HEADERIBUF_TREETRX_SYS

FIRST_RSEGDICT_HDR

Data Dict.

SYS_COLUMNSSYS_INDEXESSYS_FIELDS

Block 1 (64 pages)Block 2 (64 pages)

iblogfile0 iblogfile1 iblogfile2

Tables withfile_per_tableDoublewrite Buffer

Buffe

r Poo

l Data Dictionary Cache

Adaptive Hash Indexes

Buffer Pool LRU

Additional Mem Pool

Log Buffer

Log

Gro

up

Buffer Pool Flush List

In Memory On Diskhttps://github.com/jeremycole/innodb_diagrams

Page 15: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted15

InnoDBTransa

ctio

nSystem

Storage

Caching

SYS_TABLES

ibda

ta1

spac

e 0

Page Cache

A.ibd

B.ibd

C.ibd

IBUF_HEADERIBUF_TREETRX_SYS

FIRST_RSEGDICT_HDR

Data Dict.

SYS_COLUMNSSYS_INDEXESSYS_FIELDS

Block 1 (64 pages)Block 2 (64 pages)

iblogfile0 iblogfile1 iblogfile2

Tables withfile_per_tableDoublewrite Buffer

Buffe

r Poo

l Data Dictionary Cache

Adaptive Hash Indexes

Buffer Pool LRU

Additional Mem Pool

Log BufferLo

g G

roup

Buffer Pool Flush List

SELECT * FROM a WHERE id = 10;

mysqld

Not Found

Query  -­‐  Non  Cached

Page 16: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted16

InnoDBTransa

ctio

nSystem

Storage

Caching

SYS_TABLES

ibda

ta1

spac

e 0

Page Cache

A.ibd

B.ibd

C.ibd

IBUF_HEADERIBUF_TREETRX_SYS

FIRST_RSEGDICT_HDR

Data Dict.

SYS_COLUMNSSYS_INDEXESSYS_FIELDS

Block 1 (64 pages)Block 2 (64 pages)

iblogfile0 iblogfile1 iblogfile2

Tables withfile_per_tableDoublewrite Buffer

Buffe

r Poo

l Data Dictionary Cache

Adaptive Hash Indexes

Buffer Pool LRU

Additional Mem Pool

Log BufferLo

g G

roup

Buffer Pool Flush List

SELECT * FROM a WHERE id = 10;

mysqld

Query  -­‐  Cached

Page 17: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted17

Update  Query  in  a    Transaction  (simplified)

InnoDBTransa

ctio

nSystem

Storage

Caching

SYS_TABLES

ibda

ta1

spac

e 0

Page Cache

A.ibd

B.ibd

C.ibd

IBUF_HEADERIBUF_TREETRX_SYS

FIRST_RSEGDICT_HDR

Data Dict.

SYS_COLUMNSSYS_INDEXESSYS_FIELDS

Block 1 (64 pages)Block 2 (64 pages)

iblogfile0 iblogfile1 iblogfile2

Tables withfile_per_tableDoublewrite Buffer

Buffe

r Poo

l Data Dictionary Cache

Adaptive Hash Indexes

Buffer Pool LRU

Additional Mem Pool

Log Buffer

Log

Gro

up

Buffer Pool Flush List

UPDATE a SET col1 = ‘new’ WHERE id = 10;

mysqld

commit;

Page 18: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Log  Files

• Provide  recovery.  • Only  written  to  in  regular  operation.  • Read  only  required  if  there  is  a  crash.  • Are  rewritten  over-­‐and-­‐over  again.  • Think  of  it  like  a  tank  tread.

18

Page 19: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted19

Program  Agenda

Introduction  

MySQL  Architecture  

Backup  &  Replication  

Performance  &  Capacity  Planning

1

2

3

4

1

3

Page 20: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Backup  (Theory)

• Cold  • Warm  • Hot

20

Page 21: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Backup  Consistency

• Must  represent  a  single  point  in  time

21

Page 22: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 22

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

invoice_item

Page 23: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 23

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

invoice_item

Page 24: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 24

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

invoice_item

Page 25: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 25

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc4 Savvy Squirrel Global

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc4 4 # Savvy Squirrels

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

6 4 Premium Acorn $999

invoice_item

Page 26: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 26

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc4 Savvy Squirrel Global

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc4 4 # Savvy Squirrels

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

6 4 Premium Acorn $999

invoice_item

Page 27: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 27

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc4 Savvy Squirrel Global

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc4 4 # Savvy Squirrels

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

6 4 Premium Acorn $999

invoice_item

Page 28: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | 28

customerid name

1 Acorn Enterprises2 First National Squirrel Bank3 Squirrels 5000 Inc4 Savvy Squirrel Global

invoiceid customerid ...

1 1 # Acorn Enterprises2 1 # Acorn Enterprises3 3 # Squirrels 5000 Inc4 4 # Savvy Squirrels

customer

invoice

itemid invoiceid Description Amount

1 1 1 hrs Professional Services

$10

2 2 Acorn Parts $12.50

3 2 5 hrs Professional Services

$50

4 3 Delivery Fee $17.50

5 3 Replacement Acorn

$20

6 4 Premium Acorn $999

invoice_item

Page 29: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Backup  Methods

• mysqldump  • Is  warm  by  default.    Can  safely  be  set  hot  if  all  InnoDB.  • Logical  backup.    Slow  to  recover.  

• Snapshot  Based  -­‐  i.e.  LVM  snapshot.  • Consistent  +  Hot  • May  have  higher  impact  during  snapshot  phase.  • Best  used  with  a  script  like  mylvmbackup.

29

Page 30: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Backup  Methods  (cont.)

• MySQL  Enterprise  Backup  • Commercial  Tool  • Hot  with  InnoDB.    Warm  for  MyISAM  phase  • Low  Impact

30

Page 31: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

How  to  Decide?

• MTTR  of  logical  backups  is  much  higher.  • Most  failures  are  accidents  (either  by  DBA  or  user).  

• “I  accidentally  updated  the  phone  number  of  the  wrong  customer!”

31

Page 32: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Replication

• Built  in  to  MySQL  • Easy  to  setup  and  get  running  • Uses  same  port  3306  • Most  deployments  will  use  Replication

32

Page 33: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Replication  Options

• Row  Format  • Statement  Based  (default)  • Row  Based  

• Global  Transaction  IDs  • Semi-­‐Synchronous  • Time  delayed

33

Page 34: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted34

Program  Agenda

Introduction  

MySQL  Architecture  

Backup  &  Replication  

Performance  &  Capacity  Planning

1

2

3

4

1

4

Page 35: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Quality  of  Service

• Often  better  to  let  one  too  demanding  user  go  unserviced  than  all  users  go  unserviced  

• Statement  Timeout  coming  to  MySQL  5.7.

35

Page 36: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Quality  of  Service  (cont.)

• Replication  Slaves,  Hadoop  very  useful.  • Important  to  Optimize  for  Response  Time  (user  facing)  before  sub  optimizing  for  background  tasks.

36

Page 37: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

ConfigurationMost  important  settings

• innodb_buffer_pool_size  -­‐  50-­‐80%  of  memory  • innodb_log_file_size  -­‐  up  to  4G  quite  safe

37

Page 38: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

ConfigurationAlso  Useful

• innodb_flush_method  • innodb_flush_neighbors  • innodb_io_capacity  and  innodb_lru_scan_depth  • Explained  here:  

• http://www.tocker.ca/2013/09/17/what-­‐to-­‐tune-­‐in-­‐mysql-­‐56-­‐after-­‐installation.html

38

Page 39: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

How  much  can  you  expect  from  config?Versus  other  optimizations

• Optimizing  Queries  is  critical  to  performance  • SQL  is  hard  

• Easy  to  write  • Hard  to  sight  check  for  performance

39

Page 40: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

How  to  find  slow  queries?

• Pro-­‐active:  • Design  or  review  all  schema  and  queries.  

• Re-­‐active:  • Find  worst  offenders  and  suggest  improvements  to  development  staff.

40

Page 41: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Realistic  Compromise

• Query  and  index  usage  changes  over  time  • You  have  to  have  tools  in  place  to  find  slow  SQL  • We  release  MySQL  Enterprise  Monitor  for  this  purpose

41

Page 42: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Hardware  Choices

• Frequently  memory  is  under-­‐provisioned  • http://rhaas.blogspot.ca/2014/08/memory-­‐matters.html  (PostgreSQL)  

• http://www.tocker.ca/2013/05/10/twice-­‐as-­‐much-­‐ram-­‐does-­‐not-­‐equal-­‐twice-­‐as-­‐fast.html  (Me;  MySQL)

42

Page 43: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Hardware  Choices  (cont.)

• SSD  transition  is  real  • But  still  better  to  buy  more  memory  first  

• Database  need  to  write  for  persistence.    Heavier  IO  requirements  than  any  other  tier

43

Page 44: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted

Hardware  Choices  (cont.)

• More  CPUs  can  mean  more  consistent  performance  • Some  level  of  scale  up  +  scale  out  is  best:  

• http://www.tocker.ca/2014/04/22/five-­‐reasons-­‐why-­‐vertical-­‐scalability-­‐matters.html  

44

Page 45: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    |

Non  Exhaustive  List  of  Duties

• Backup  • Replication  • Patching  &  Upgrades  • Performance  and  Capacity  Planning  • Coaching  &  Mentoring  • Fighting  Fires

45

Page 46: MySQL For Linux Sysadmins

Copyright  ©  2014  Oracle  and/or  its  affiliates.  All  rights  reserved.    | Oracle  Confidential  –  Internal/Restricted/Highly  Restricted46

Page 47: MySQL For Linux Sysadmins