Upload
technical-dude
View
3.926
Download
4
Tags:
Embed Size (px)
Citation preview
Presented by,MySQL AB® &
O’Reilly Media, Inc.
Best Practices For Deploying MySQL on Solaris
Ritu KambojJenny Chen
Agenda
MySQL -Solaris Integration MySQL High Availability Data Service Consolidate MySQL Deployment Using Solaris
Containers DTrace (Jenny Chen)
MySQL-Solaris Integration
Optimized MySQL on Open Solaris
MySQL 5.0.45 (32bit) integrated with Open Solaris build 76 SXDE 01/08
MySQL 5.0.45 (64bit) integrated with Open Solaris build 87 SXCE
Layout of MySQL on Open Solaris
MySQL 5.0.45 packages Default data directory
/var/mysql/5.0/data Default configuration directory
/etc/mysql/5.0 Installation directory
/usr/mysql/5.0/
Latest version accessible from /usr/mysql/bin Symbolic link from all directories involved in 5.0.45 release to corresponding /usr/mysql
Optimization Of MySQL On Open Solaris
Compiled with optimal compiler options using Sun Studio compilers
Added SMF support for MySQL Easily accommodates varying configurations Initializes MySQL database thereby enhancing user experience Mysql user can manage MySQL database via SMF
Support for MySQL cluster engine (ndb)
Compiler Optimization
Enable in-lining Change header file univ.i to add Sun Studio
If (!defined(SUNPRO_C) #define UNIV_MUST_NOT_INLINE
Enable pre-fetching -xprefetch=auto and -xprefetch_level=3
Set optimization level -xO4
Compiler Optimization (Contd)
Feedback optimization (not yet implemented) Workload is key -xprofile -xipo Interprocess optimization About 10% improvement
Link with libmtmalloc Library for threaded application About 8% improvement
Service Management Facility (SMF)
Makes Solaris Services Self-Healing Services automatically restart in dependency
order Misbehaving and mis-configured services are
easier to debug Log files for each service
Administrators can securely delegate tasks to non-root users
MySQL-Service Management Facility
Dynamically configured properties mysql/bin
Installation , default : /usr/mysql/5.0/bin mysql/data
Data Directory , Default: /var/mysql/5.0/data mysql/enable_64bit
Flag to select 32bit or 64bit , Default : false
First time installations Creates system tables (mysql_install_db.sh )
Starting MySQL on Open Solaris
Older Open Solaris Build (Build 76- Build-87) 32Bit MySQL integrated Create mysql user Enable MySQL SMF service
Latest Open Solaris Build (Build 87 onwards) 32Bit and 64bit MySQL integrated Default mode : 32bit service For starting 64bit
Set enable_64bit == true
Optimized MySQL on Solaris 10
CoolStack SAMP stack + more
Optimized MySQL download Standalone package SAMP stack component
Available at Sun Download center: http://cooltools.sunsource.net/coolstack
Version MySQL 5.0.45 Similar optimizations as in Open Solaris
MySQL-High Availability Data Service
Solaris Cluster Overview Provides general purpose HA platform
Fifty percent of enterprises that lack arecovery plan go out of business
within one year of a significant disaster
Gartner Group
Availability is our customers' most critical requirement
Sun Cluster VOC Survey
Solaris Cluster : Hardware Components Servers with local storage
Can have up to 16 nodes Shared storage
Tolerates single-node failures Centralizes configuration files
Cluster interconnect At least two redundant networks
Public network interfaces Spreads outbound packets
Solaris Cluster Algorithms Cluster membership monitor
Ensures data integrity Determines cluster membership
Cluster configuration repository Global repository Ensures consistent view
Disk Fencing Fences off non-cluster nodes Prevents Partition
Quorum Uses a majority voting schema
MySQL High Availability Data Service HA-MySQL is a failover data service
Nod
e 1
Nod
e 2
Nod
e 3
Nod
e 4
DDBB
MySQL High Availability Data Service Supported configurations
Standalone MySQL server MySQL replication server
Single/Multiple MySQL instances in master configuration
Single/Multiple MySQL instances in slave configuration
Solaris containers supportGlobal zone Non-global failover zone Non-global zone
HA-MySQL Service Deployment
Node1 Node2:Zone2Node1:Zone1
Node2
MySQL
Stor Host
MySQL
Stor Host
MySQL-Solaris Cluster Benefits Enhanced end-to-end infrastructure availability Continuous MySQL Availability
Automatic failover if master node fails
Low cost solution Software is free and open sourced
Efficient Resource Utilization Multiple applications can be consolidated
Ease of operations SC enables clustered systems to be managed as if they were on a single system
Additional information
Step by step deployment guide http://docs.sun.com/app/docs/doc/819-3059
Failover study of HA-MySQL http://blogs.sun.com/krishs/date/200804
Solaris Cluster http://www.sun.com/software/solaris/cluster/index.xmlhttp://opensolaris.org/os/community/ha-clusters/
Consolidate MySQL installations using Solaris Containers
Solaris Containers Containers : Zones + Resource Management
Zones: isolated virtual application environments Resource management – resource control (CPU, Memory)
Achieving Consolidation Goals Reduce Hardware
Combine low utilization systems
Isolate applications from faults Maintain Service Levels
Fine tune response times
MySQL Consolidation Study System Configuration
Sun Fire X4100 (4 CPU, 8 GB Memory)
Local container configuration ( 1 CPU , 1 GB Memory
Sysbench Read-only Sysbench read-write
32 64 128 256
1300
1400
1500
1600
1700
1800
1900Sysbench read-only
No Containers
With Containers
Number of threads
Th
rou
gh
pu
t
32 64 128 256
0
100
200
300
400
500
600
700Sysbench read-write
No Containers
With Containers
Number of threads
Th
rou
gh
pu
t
Dtrace (Jenny Chen)
DTrace
Use DTrace with MySQL to drill down Monitoring MySQL Performance
Examples & Solutions
Easy Steps To add DTrace probes into MySQL core server and storage Engines
Easy Steps to display MySQL DTrace probes into Chime visualization Tool for DTrace
Why DTrace
Solaris 10 Dynamic Tracing Facility to provide comprehensive view of operating system and application behaviour
> DTrace to examine particular system areas: disk I/O, CPU, Memory
> Process Tracing and Debugging
USDT(User-level statically defined tracing) place custom probes in application code
Add USDT into MySQL source to monitor MySQL and gather the useful data missing by the current MySQL monitor tools:
DTrace: Monitor I/Os Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)
#!/usr/sbin/dtrace –s
#pragma D option quiet
io:::wait-start
/ execname == “mysqld” /
{ self->start = timestamp; }
io:::wait-done
/ execname == “mysqld” && self->start /
{ this->elapsed = timestamp - self->start;
@files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed);
self->start = 0;
}
profile:::tick-5s
{ printf(“-------------------------------------------------\n”);
printf(“%6s %8s %20s %50s\n”, “PID”, “TIME”, “DEVICE”, “FILE”);
printa(“%6d %@8d %20s %8s \n”, @files);
printf (“------------------------------------------------\n”); }
#./mysqliowait.d
---------------------------------------------------
PID TIME DEVICE FILE
113 234 /dev/dsk/c0t2d0s0 /usr/local/mysql/data/ibdata1
DTrace: Monitor CPU classic performance problem
#!/usr/sbin/dtrace -s#pragma D option quietsyscall:::entry{
self->ts=vtimestamp;}syscall:::return/self->ts/{
@a[execname, probefunc] = count(); @b[execname, probefunc] = sum (vtimestamp - self->ts); self->ts=0;}END {
printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","COUNT"); printa("%-16s %-16s %-@8d\n",@a); printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","TIME"); printa("%-16s %-16s %-@8d\n",@b);
}
# ./syscall.d EXEC SYSCALL COUNT devfsadm lwp_park 1 dtrace fstat 1 ... mysqld read 106542 mysqld gtime 109613 mysqld pread 1181669 --------------------------------------------------- EXEC SYSCALL TIME dtrace lwp_sigmask 218 pkill getpid 302 ... mysqld read 259284183 mysqld write 267556239 mysqld pread 4650457224
Agrregated user stack backtrace to understand of the nature of pread() in MySQL source code – useful for mysql developers
# dtrace -n 'syscall::pread:entry / execname == "mysqld" / { @[ustack()]=count() } dtrace: description 'syscall::pread:entry ' matched 1 probe ... libc.so.1`_pread+0xa mysqld`my_pread+0x54 mysqld`_mi_read_static_record+0x67 mysqld`mi_rnext+0x1fe ... mysqld`handle_one_connection+0x855 libc.so.1`_thr_setup+0x67 1564811 .... Replace high cost pread with “--myisam_use_mmap=1”- useful for mysql DBA
A 94% performance improvement !
Dtrace: Memory Analysis Check which process causes anonymous page in
# dtrace -n anonpgin '{@[execname] = count()}' dtrace: description anonpgin matched 1 probe sshd 2 vmstat 23 mysqld 673
Use Dtrace to measure waiting for paging in# ./ whospaging.d – available at Solaris Internals(http://www.solarisinternals.com/si/dtrace/) Who's on cpu (milliseconds): sshd 1 vmstat 3 mysqld 120 sched 43210Who's waiting for pagin (milliseconds): mysqld 239082
Dtrace Probes In MySQL Provide deep view of internal MySQL core server and storage engines'
operation & behaviour>Database information>Query execution latency>Index & table scan cost>Wait events inside MyISAM & Innodb>Deadlock information>Query cache hit/miss>And many more...
Speed resolution of performance bottlenecks with in database design and MySQL server configurations
Negligible performance overhead
Easy steps to create & insert your own Dtrace probes into MySQL
MySQL DTrace GUI Monitor Tool - Chime
Query execution Time
Enable Slow query log with “–log-slow-queries” requires re-start MySQL server
SQL statements with query execution time longer than “long_query_time” second in the log file
SQL statements generating most loads on the application may not in slow query log
Replication query statements are not available in slow query log
Time spending by the query optimizer to generate query plan is not available in slow query log
Using DTrace can get mising query execution information online
Insert DTrace Probes Into MySQL
Step 1: Figure out what probes are needed to insert into the source code
Step 2: Define MySQL Provider and probes
# cat mysql_dtrace.d
provider mysql
{
probe query__execute__start(void *, char *, char *, const char *, char *);
probe query__execute__finish(void *, char *, char *, const char *, char *,int);
}
>Two Probes defined in the mysql provider>Note to use two underscore(__) translated to hypen automatically
Step 3: Define a header file “mysql_dtrace.h” with definitions for probes
dtrace -h -s mysql_dtrace.d
#ifndef _MYSQL_H
#define _MYSQL_H
#define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4) \
__dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)
#define DTRACE_QUERY_EXECUTE_START_ENABLED() \
__dtraceenabled_mysql___query__execute__start()
extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char *, char *);
extern int __dtraceenabled_mysql___query__execute__start(void);
#endif
Step 4: Insert the probes into source code #include <mysql_dtrace.h>
...
bool mysql_execute_command(THD *thd)
{
DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db,
thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query);
...
DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db,
thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,
thd->query, res ==0 ? 0: -1);
go to end;
...
}
Step 5: Build MySQL with DTrace In the Makefile.in, compile 64-bit MySQL with Dtrace
mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)
mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)
dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)
Inserting DTrace probes comleted, DTrace probes are ready to use!
Step 6: Use inserted DTrace probes to measure query execution time with other database information #!/usr/sbin/dtrace -s
…
mysql*:::query-execute-start
{ self->start = timestamp; }
mysql*:::query-execute-finish
/self->start/
{ this->query = copyinstr(arg4); }
mysql*:::query-execute-finish
/self->start/
{ this->elapsed = (timestamp - self->start) / 1000000;
this->who = strjoin(copyinstr(arg2), strjoin("@", copyinstr(arg3)));
printf(" %-16.16s %-18.18s %5d %3d %-32.32s\n", arg1 ? copyinstr(arg1)
: ".", this->who, this->elapsed, (int)arg5, this->query); self->start = 0;
}
# ./mysqld_qestat.d
DATABASE USER@HOST ms RET QUERY
sbtest root@localhost 0 0 show tables
sbtest root@localhost 0 0 show databases sbtest root@localhost 178 0 select * from sbtest
Use the same steps to insert Dtrace probes to <source_tree>/sql/sql_select.cc at the start and end of choose_plan() function to measure the time spent in query optimization
optimizer_prune_level=1 reduce query compilation time Reduce optimizer_search_depth or optimizer_search_depth=0
Index & Table scan cost
Identify the places in MySQL source to handle scanning index, and table>Index-scan functions: index_next, index_next_same, index_prev, index_first, index_last>Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos>Insert DTrace Probes at the start and before return from the functions can measure the time spending on scanning table or index.
mysql*:::innodb-index-next-start
{ @indexnext[args[0]] = count();
self->inext = timestamp; }
mysql*:::innodb-index-next-finish
/self->inext/
{ @indexnexttime[args[0]] = sum(timestamp - self->inext);
self->inext = 0; }
Expensive index-scan/table-scan report from Dtrace requires to optimize schema accordingly
Buffer wait in Innodb Innodb buffer wait is common in I/O-bound MySQL system while reading
page synchronous from disk Insert Dtrace probes at: innobase/buf/buf0rea.c,
ulint buf_read_page( ulint space, ulint offset) { ... DTRACE_INNODB_BUFFER_WAIT_START(); /* We do the i/o in the synchronous aio mode to save thread switches: hence TRUE */ count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space, tablespace_version, offset); DTRACE_INNODB_BUFFER_WAIT_FINISH(); ... }
Increase innodb_buffer_pool size Tune SQL to access rows with fewer block reads(i.e. By adding indexes)
Query cache probes
Use DTrace to measure query cache hit and query cache miss to determine how well the query cache is performing
> Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client
> DTrace Test script to report query cache hit and miss counts by query string, and totals
mysql*:::query-cache-hit,
mysql*:::query-cache-miss
{ this->query = copyinstr(arg4); }
mysql*:::query-cache-hit
{ @elapsed[this->query, "hit"] = count();
hits++; }
mysql*:::query-cache-miss
{ @elapsed[this->query, "miss"] = count();
misses++; }
# ./mysqld_qchit.d
Tracing... Hit Ctrl-C to end.
^C
QUERY RESULT COUNT
select * from months miss 1
select * from months where num > 3 hit 1
select * from months where num > 3 and num < 9 miss 1
show databases miss 1
show tables miss 1
select * from months hit 9
Hits : 10
Misses : 4
Hit Rate : 71%
Tuning “query_cache_size” variable according to the hit/miss rate
DTrace Performance Impact
Inserting DTrace Probes into MySQL source code are useful for MySQL DBA, MySQL & application developers. The performance impact of adding DTrace probes is critical for enterprise environment.
Cost of inserting USDT probe can be basically negligible: Each probe inserted into the source code can be enabled by adding the code like:
if (PROVIDER_PROBE_ENABLED() { PROVIDER_PROBE(arg0,...); }
DTrace Probes In MySQL 6.0
probe insert_row_start();probe insert_row_end();probe filesort_start();probe filesort_end();probe delete_start();probe delete_end();probe select_start()probe select_end();probe update_start();probe update_end();
compile with “--enable-dtrace” configure option to use the Dtrace probes
Integrate with Chime Tool Chime is graphical tool for visualizing DTrace aggregations. It
provides alternative CLI-based tool output with more visually
apprealing and more useful to display data over time
Available to download: http://www.opensolaris.org/os/project/dtrace-chime. One-step installation:
> Run pkgadd -d osol0chime-<arch>-1.4.pkg
chime requires Solaris Nevada build 35 or later
> Run local: /opt/OSOL0chime/bin/chime
> Run remotely: /opt/OSOL0chime/bin/chime <hostname>
chime provides wizard to automatically generate new display for DTrace script
Resources > http://www.opensolaris.org/os/community/dtrace - OpenSolaris Community: Dtrace > http://developers.sun.com/solaris/articles/solaris_perftools.html - Solaris Performance and Tools > http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view - Statically Defined Tracing for User Applications chapter of DTrace Manual > http://www.brendangregg.com - DTrace toolkits > http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases - DTrace Topics Databases Acknowledgements Brandan Gregg – Sun Microsystems Engineer in Advanced Products Group