Upload
ashokteam
View
225
Download
0
Embed Size (px)
Citation preview
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 1/49
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 2/49
Agenda
MySQL -Solaris IntegrationMySQL High Availability Data ServiceConsolidate MySQL Deployment Using SolarisContainersDTrace (Jenny Chen)
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 3/49
MySQL-Solaris Integration
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 4/49
Optimized MySQL on Open Solaris
MySQL 5.0.45 (32bit) integrated with OpenSolaris build 76
SXDE 01/08MySQL 5.0.45 (64bit) integrated with OpenSolaris build 87
SXCE
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 5/49
Layout of MySQL on Open Solaris
MySQL 5.0.45 packages Default data directory
/var/mysql/5.0/dataDefault configuration directory
/etc/mysql/5.0Installation directory/usr/mysql/5.0/
Latest version accessible from /usr/mysql/binSymbolic link from all directories involved in 5.0.45release to corresponding /usr/mysql
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 6/49
Optimization Of MySQL On Open Solaris
Compiled with optimal compiler options usingSun Studio compilersAdded SMF support for MySQL
Easily accommodates varying configurationsInitializes MySQL database thereby enhancing user
experienceMysql user can manage MySQL database via SMF
Support for MySQL cluster engine (ndb)
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 7/49
Compiler Optimization
Enable in-lining Change header file univ.i to add Sun Studio
If (!defined(SUNPRO_C)#define UNIV_MUST_NOT_INLINE
Enable pre-fetching-xprefetch=auto and -xprefetch_level=3
Set optimization level-xO4
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 8/49
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 9/49
Service Management Facility (SMF)
Makes Solaris Services Self-HealingServices automatically restart in dependencyorder
Misbehaving and mis-configured services areeasier to debug
Log files for each serviceAdministrators can securely delegate tasks tonon-root users
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 10/49
MySQL-Service Management Facility
Dynamically configured properties mysql/bin
Installation , default : /usr/mysql/5.0/bin mysql/data
Data Directory , Default: /var/mysql/5.0/data mysql/enable_64bit
Flag to select 32bit or 64bit , Default : false
First time installations Creates system tables (mysql_install_db.sh )
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 11/49
Starting MySQL on Open Solaris
Older Open Solaris Build (Build 76- Build-87) 32Bit MySQL integrated Create mysql user Enable MySQL SMF service
Latest Open Solaris Build (Build 87 onwards) 32Bit and 64bit MySQL integrated Default mode : 32bit serviceFor starting 64bit Set enable_64bit == true
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 12/49
Optimized MySQL on Solaris 10
CoolStack SAMP stack + more
Optimized MySQL downloadStandalone packageSAMP stack component
Available at Sun Download center: http://cooltools.sunsource.net/coolstack
Version MySQL 5.0.45Similar optimizations as in Open Solaris
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 13/49
MySQL-High Availability Data Service
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 14/49
Solaris Cluster OverviewProvides general purpose HA platform
Fifty percent of enterprises that lack arecovery plan go out of business withinone year of a significant
disaster Gartner Group
Availability is our customers'most critical requirement Sun Cluster VOC Survey
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 15/49
Solaris Cluster : Hardware ComponentsServers with local storage
Can have up to 16 nodesShared storage
Tolerates single-node failures
Centralizes configuration files
Cluster interconnectAt least two redundant networks
Public network interfaces Spreads outbound packets
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 16/49
Solaris Cluster AlgorithmsCluster membership monitor Ensures data integrity
Determines cluster membershipCluster configuration repository
Global repositoryEnsures consistent view
Disk FencingFences off non-cluster nodes
Prevents PartitionQuorum
Uses a majority voting schema
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 17/49
MySQL High Availability Data ServiceHA-MySQL is a failover data service
Node1
Node2
Node
3
Node4
DDBB
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 18/49
MySQL High Availability Data ServiceSupported configurations
Standalone MySQL server
MySQL replication server
Single/Multiple MySQL instances in master configuration
Single/Multiple MySQL instances in slave configuration
Solaris containers supportGlobal zoneNon-global failover zoneNon-global zone
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 19/49
HA-MySQL Service Deployment
Node1 Node2:Zone2 Node1:Zone1 Node2
MySQL
Stor Host
MySQL
Stor Host
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 20/49
MySQL-Solaris Cluster BenefitsEnhanced end-to-end infrastructure availabilityContinuous MySQL Availability
Automatic failover if master node fails
Low cost solution Software is free and open sourced
Efficient Resource Utilization Multiple applications can be consolidated
Ease of operations SC enables clustered systems to be managed as if
they were on a single system
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 21/49
Additional information
Step by step deployment guide http://docs.sun.com/app/docs/doc/819-3059
Failover study of HA-MySQL http://blogs.sun.com/krishs/date/200804
Solaris Cluster http://www.sun.com/software/solaris/cluster/index.xmlhttp://opensolaris.org/os/community/ha-clusters/
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 22/49
Consolidate MySQL installations using SolarisContainers
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 23/49
Solaris ContainersContainers : Zones + Resource Management
Zones: isolated virtual application environmentsResource management – resource control (CPU,
Memory)
Achieving Consolidation Goals Reduce Hardware
Combine low utilization systems
Isolate applications from faults Maintain Service Levels
Fine tune response times
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 24/49
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 25/49
Dtrace (Jenny Chen)
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 26/49
DTrace
Use DTrace with MySQL to drill downMonitoring MySQL Performance
Examples & Solutions
Easy Steps To add DTrace probes intoMySQL core server and storage Engines
Easy Steps to display MySQL DTraceprobes into Chime visualization Tool for DTrace
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 27/49
Why DTrace
Solaris 10 Dynamic Tracing Facility to providecomprehensive view of operating system andapplication behaviour
> DTrace to examine particular system areas: disk I/O, CPU,
Memory> Process Tracing and Debugging
USDT (User-level statically defined tracing) place custom probes inapplication codeAdd USDT into MySQL source to monitor MySQL and gather the useful data missing by the current MySQL monitor tools:
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 28/49
DTrace: Monitor I/Os
Exam I/O wait time by filename and mysqld(Available at DTraceToolkit)#!/usr/sbin/dtrace –s
#pragma D option quiet
io:::wait-start
/ execname == “mysqld” /
{ self->start = timestamp; }
io:::wait-done
/ execname == “mysqld” && self->start /
{ this->elapsed = timestamp - self->start;
@files[pid,args[1]->dev_pathname, args[2]->fi_pathname] = sum(this->elapsed);
self->start = 0;
}
profile:::tick-5s
{ printf(“-------------------------------------------------\n”);
printf(“%6s %8s %20s %50s\n”, “PID”, “TIME”, “DEVICE”, “FILE”);
printa(“%6d %@8d %20s %8s \n”, @files);
printf (“------------------------------------------------\n”); }
#./mysqliowait.d
---------------------------------------------------
PID TIME DEVICE FILE
113 234 /dev/dsk/c0t2d0s0 /usr/local/mysql/data/ibdata1
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 29/49
DTrace: Monitor CPUclassic performance problem#!/usr/sbin/dtrace -s#pragma D option quietsyscall:::entry{
self->ts=vtimestamp;}syscall:::return
/self->ts/{
@a[execname, probefunc] = count();@b[execname, probefunc] = sum (vtimestamp - self->ts);self->ts=0;
}END{
printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","COUNT");printa("%-16s %-16s %-@8d\n",@a);printf("%-16s %-16s %-8s\n","EXEC","SYSCALL","TIME");printa("%-16s %-16s %-@8d\n",@b);
}
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 30/49
# ./syscall.dEXEC SYSCALL COUNTdevfsadm lwp_park 1dtrace fstat 1...
mysqld read 106542mysqld gtime 109613mysqld pread 1181669 ---------------------------------------------------EXEC SYSCALL TIME
dtrace lwp_sigmask 218pkill getpid 302...mysqld read 259284183mysqld write 267556239
mysqld pread 4650457224
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 31/49
Agrregated user stack backtrace to understand of the nature of pread() inMySQL source code – useful for mysql developers
# dtrace -n 'syscall::pread:entry / execname == "mysqld" / { @[ustack()]=count() }
dtrace: description 'syscall::pread:entry ' matched 1 probe...
libc.so.1`_pread+0xamysqld`my_pread+0x54mysqld`_mi_read_static_record+0x67mysqld`mi_rnext+0x1fe
...mysqld`handle_one_connection+0x855libc.so.1`_thr_setup+0x67
1564811....
Replace high cost pread with
“--myisam_use_mmap=1”- useful for mysql DBA
A 94% performance improvement!
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 32/49
Dtrace: Memor y AnalysisCheck which process causes anonymous page in# dtrace -n anonpgin '{@[execname] = count()}'dtrace: description anonpgin matched 1 probesshd 2
vmstat 23mysqld 673
Use Dtrace to measure waiting for paging in
# ./ whospaging.d – available at Solaris Internals( http://www.solarisinternals.com/si/dtrace/ )Who's on cpu (milliseconds):
sshd 1vmstat 3mysqld 120sched 43210
Who's waiting for pagin (milliseconds):mysqld 239082
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 33/49
Dtrace Probes In MySQL
Provide deep view of internal MySQL core server and storage engines'operation & behaviour >Database information>Query execution latency>Index & table scan cost>Wait events inside MyISAM & Innodb>Deadlock information>Query cache hit/miss>And many more...
Speed resolution of performance bottlenecks with in database designand MySQL server configurations
Negligible performance overhead
Easy steps to create & insert your own Dtrace probes into MySQL
MySQL DTrace GUI Monitor Tool - Chime
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 34/49
Quer y execution Time
Enable Slow query log with “–log-slow-queries” requires re-startMySQL server
SQL statements with query execution time longer than“long_query_time” second in the log file
SQL statements generating most loads on the application maynot in slow query log
Replication query statements are not available in slow query log
Time spending by the query optimizer to generate query plan isnot available in slow query log
Using DTrace can get mising query execution information online
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 35/49
Inser t DTrace Probes Into MySQL
Step 1: Figure out what probes are needed to insert into thesource code
Step 2: Define MySQL Provider and probes# cat mysql_dtrace.d
provider mysql
{
probe query__execute__start(void *, char *, char *, const char *, char *);
probe query__execute__finish(void *, char *, char *, const char *, char *,int);
}
>Two Probes defined in the mysql provider >Note to use two underscore(__) translated to hypen automatically
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 36/49
Step 3: Define a header file “mysql_dtrace.h” with definitions for probes
dtrace -h -s mysql_dtrace.d
#ifndef _MYSQL_H
#define _MYSQL_H
#define DTRACE_QUERY_EXECUTE_START(arg0, arg1, arg2, arg3, arg4) \
__dtrace_mysql___query__execute__start(arg0, arg1, arg2, arg3, arg4)
#define DTRACE_QUERY_EXECUTE_START_ENABLED() \
__dtraceenabled_mysql___query__execute__start()
extern void __dtrace_mysql___query__execute__start(void *, char *, char *, char *, char *);
extern int __dtraceenabled_mysql___query__execute__start(void);
#endif
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 37/49
Step 4: Insert the probes into source code #include <mysql_dtrace.h>
... bool mysql_execute_command(THD *thd)
{
DTRACE_QUERY_EXECUTE_START((void *)thd, thd->db,
thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,thd->query);
...
DTRACE_QUERY_EXECUTE_FINISH((void*)thd, thd->db,
thd->security_ctx->user, (char *)thd->security_ctx->host_or_ip,
thd->query, res ==0 ? 0: -1);
go to end;...
}
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 38/49
Step 5: Build MySQL with DTrace In the Makefile.in, compile 64-bit MySQL with Dtrace
mysqld_OBJECTS = $(am_mysqld_OBJECTS) mysql_dtrace.$(OBJEXT)
mysql_dtrace.o:$(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)
dtrace -G 64 -s $(top_srcdir)/include/mysql_dtrace.d $(am_mysqld_OBJECTS)
Inserting DTrace probes comleted , DTrace probes are ready to use!
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 39/49
Step 6: Use inserted DTrace probes to measure query executiontime with other database information
#!/usr/sbin/dtrace -s
…mysql*:::query-execute-start
{ self->start = timestamp; }
mysql*:::query-execute-finish
/self->start/
{ this->query = copyinstr(arg4); }
mysql*:::query-execute-finish
/self->start/
{ this->elapsed = (timestamp - self->start) / 1000000;
this->who = strjoin(copyinstr(arg2), strjoin("@", copyinstr(arg3)));
printf(" %-16.16s %-18.18s %5d %3d %-32.32s\n", arg1 ? copyinstr(arg1)
: ".", this->who, this->elapsed, (int)arg5, this->query); self->start = 0;
}
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 40/49
# ./mysqld_qestat.d
DATABASE USER@HOST ms RET QUERY
sbtest root@localhost 0 0 show tablessbtest root@localhost 0 0 show databasessbtest root@localhost 178 0 select * from sbtest
Use the same steps to insert Dtrace probes to<source_tree>/sql/sql_select.cc at the start and end of choose_plan()function to measure the time spent in query optimization
optimizer_prune_level=1 reduce query compilation time
Reduce optimizer_search_depth or optimizer_search_depth=0
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 41/49
Index & Table scan cost
Identify the places in MySQL source to handle scanning index, andtable>Index-scan functions: index_next, index_next_same, index_prev,index_first, index_last>Table-scan functions: rnd_init, rnd_end, rnd_next, rnd_pos>Insert DTrace Probes at the start and before return from the functions
can measure the time spending on scanning table or index.mysql*:::innodb-index-next-start{ @indexnext[args[0]] = count();
self->inext = timestamp; }mysql*:::innodb-index-next-finish
/self->inext/{ @indexnexttime[args[0]] = sum(timestamp - self->inext);self->inext = 0; }
Expensive index-scan/table-scan report from Dtrace requires tooptimize schema accordingly
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 42/49
Buffer wait in Innodb Innodb buffer wait is common in I/O-bound MySQL system while reading
page synchronous from disk Insert Dtrace probes at: innobase/buf/buf0rea.c,
ulint buf_read_page( ulint space, ulint offset){ ...
DTRACE_INNODB_BUFFER_WAIT_START();
/* We do the i/o in the synchronous aio mode to save threadswitches: hence TRUE */count2 = buf_read_page_low(&err, TRUE, BUF_READ_ANY_PAGE, space,
tablespace_version, offset);DTRACE_INNODB_BUFFER_WAIT_FINISH();
...}
Increase innodb_buffer_pool size Tune SQL to access rows with fewer block reads(i.e. By adding indexes)
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 43/49
Query cache probes
Use DTrace to measure query cache hit and query cache miss todetermine how well the query cache is performing > Insert DTrace Probes at: sql/sql_cache.cc function: send_result_to_client
> DTrace Test script to report query cache hit and miss counts by query string, andtotals
mysql*:::query-cache-hit,
mysql*:::query-cache-miss
{ this->query = copyinstr(arg4); }
mysql*:::query-cache-hit { @elapsed[this->query, "hit"] = count();
hits++; }
mysql*:::query-cache- miss{ @elapsed[this->query, "miss"] = count();
misses++; }
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 44/49
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 45/49
DTrace Performance Impact
Inserting DTrace Probes into MySQL source code are useful for MySQL DBA, MySQL & application developers. Theperformance impact of adding DTrace probes is critical for enterprise environment.
Cost of inserting USDT probe can be basically negligible:
Each probe inserted into the source code can be enabled by adding thecode like:
if (PROVIDER_PROBE_ENABLED(){
PROVIDER_PROBE(arg0,...);}
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 46/49
DTrace Probes In MySQL 6.0
probe insert_row_start(); probe insert_row_end(); probe filesort_start(); probe filesort_end(); probe delete_start(); probe delete_end(); probe select_start() probe select_end();
probe update_start(); probe update_end();
compile with “--enable-dtrace” configure option to usethe Dtrace probes
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 47/49
Integrate with Chime Tool Chime is graphical tool for visualizing DTrace aggregations. It
provides alternative CLI-based tool output with more visuallyapprealing and more useful to display data over time
Available to download:http://www.opensolaris.org/os/project/dtrace-chime . One-stepinstallation: > Run pkgadd -d osol0chime-<arch>-1.4.pkg
chime requires Solaris Nevada build 35 or later > Run local: /opt/OSOL0chime/bin/chime > Run remotely: /opt/OSOL0chime/bin/chime <hostname>
chime provides wizard to automatically generate new display for DTrace script
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 48/49
8/8/2019 Best Practices for Deploying MySQL on the Solaris Platform
http://slidepdf.com/reader/full/best-practices-for-deploying-mysql-on-the-solaris-platform 49/49
Resources> http:// www.opensolaris.org / os/community/dtrace
- OpenSolaris Community: Dtrace
> http://developers.sun.com/solaris/articles/solaris_perftools.html - Solaris Performance and Tools > http://docs.sun.com/app/docs/doc/817-6223/6mlkidlms?a=view
- Statically Defined Tracing for User Applications chapter of DTraceManual
> http://www.brendangregg.com- DTrace toolkits
>http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Databases
- DTrace Topics Databases
Acknowledgements Brandan Gregg – Sun Microsystems Engineer in Advanced ProductsGroup