1

Satyajit TripathiISV-Engineering, Sun Microsystems

SOLARISTM DTRACEINTRODUCTION

1

2

Audience

• Chief Executive Officer (CXO, CTO)• Software Developer• System Administrator• Support and Service Engineer• Rest Of The World

3

Agenda

• Troubleshooting Scenarios• What is SolarisTM DTrace• DTrace Architecture and Framework• DTrace Components (Need-To-Know)• Additional Privileges for DTrace (How-To)• Useful Links and Resources• Exercises with DTrace (Touch 'n' Feel)• Open Forum

4

Troubleshooting Scenarios

• Fatal, Reproducible and Test Case available> Traditional debugging techniques. Provide patch.

• Fatal, Non Reproducible> Crash dump, Analyze Core. Use mdb(1), dbx(1)> Postmortem, Speculate Static Snapshot. Iterative

process of providing Debug binary, studying log files etc.

• Non Fatal, Transient Failure and Unacceptable QoS> Traditional techniques of using truss(1), mdb(1)> Challenges, when a dynamic problem detection requires

Invasion, Embedding or Aggregation in production> Here, DTrace can be the Savior on Solaris10 or above

5

What Is SolarisTM DTrace

• Dynamic tracing facility framework for Solaris10• Unique in its focus on Production system, and its

integration of User-level and Kernel-level tracing• Allows tracing of arbitrary data and arbitrary

expression using the D Language• Overhead nearly 0% on System Performance• 50,000+ Probes available on a System by default• No re-compilation of application required• Safe, Powerful, Flexible, and Easy-to-Use• DTrace is Open Source under CDDL License

6

Dynamic Tracing (D Trace)

• A Comprehensive Framework for SolarisTM Operating System Environment> To Implement New DTrace Providers> To Implement fully Configurable DTrace Probes> To Implement New DTrace Consumers and Data Display

• Observability with DTrace> Aggregate arbitrary behavior of the OS and User programs> Dynamically Enable and Manage Probes> Dynamically Associate Predicates and Actions with Probes> Dynamically Manage Trace Buffers and Probe Overhead> Examine Live Production System or a Crash Dump

7

DTrace Architecture and Framework

Probes

• Need To Know> D Language> DTrace Probes> DTrace Provider> Actions for Probes> DTrace Consumer

8

D Language• 'D' Language is like 'C' and Constructs similar to awk

• Based on Actions and Predicates

• Access to Global, Thread local and Probe local variables

• Rich built-in Variable set. Associative Arrays

• Access to global Kernel variables and structures

• Support for ANSI-C Operators and Data Aggregation

• Example : Trace the pid of process “date” with syscall open(2)

#!/usr/sbin/dtrace -s

syscall::open:entry

/execname == “date”/

{trace(pid);

}

Probe Name o Entry point of open

Predicate o Process is named “date”

Action o Print the process ID

9

DTrace Probe • Defined by point of Instrumentation within OS Kernel

• Probe has a Name, Identifies the Function and Module it Instruments, Accessible through Provider

• Four Attributes: Provider, Module, Function & Name, defines a tuple to uniquely identify a Probe

probe description (provider:module:function:name)/ predicate /

{

action statements}

• Each Probe is Assigned an Integer Identifier

• Example : Type command dtrace -l

10

DTrace Provider

• A Methodology for Instrumenting Probes

• Provider registers Probes in the system

• Provider is informed by DTrace to Enable a Probe, and transfers the Control of the Probe to the DTrace

• Verify Providers : dtrace -l | moreID PROVIDER MODULE FUNCTION NAME1 dtrace BEGIN2 dtrace END3 dtrace ERROR...Output truncated

• Example of Providers : > syscall: Traces syscalls fbt : Traces in-kernel functions> pid : Traces application functions sched: Traces scheduling events> io : Traces system IO proc : Process/Thread creation, term, SIG

11

Actions for Probes

• Actions can be taken when Probe is triggered• Actions are Programmable (Very useful feature)• Most Actions record a specified State of the system• Expressions in D Language are acceptable as

Action Parameter(s)

12

DTrace Consumer

• Process that interacts with DTrace, could be a Command line or a Script (script.d)

• DTrace handles multiplexing, Supports concurrent Consumers

• dtrace(1M) is DTrace Consumer

• D Script Construction: Probe description, Predicate and ActionCreate Filename : myscript.d as below

#!/usr/sbin/dtrace -ssyscall::write:entry

/ execname == “bash”/

{ printf(“bash with pid %d called write system call\n”,pid);

}

Run the myscript.d as below

# dtrace -s myscript.d

dtrace: script 'myscript.d' matched 1 probe

13

Additional Privileges for DTrace• All Privileges for superuser root and

• Selective Privileges for non root user

• Privilege Groups provide Selective Access> dtrace_user : Provider syscall and profile> dtrace_proc : Provider pid and usdt> dtrace_kernel: Provider fbt and Kernel data structures> proc_owner : *Probe others Process which has sub-set privilege(s)

NOTE : You can observe only those processes for which you have the privilege, makes it Safe!

14

Privileges How-To

• Temporary Privilege to a running Process with PID 2596> Command ppriv -s A+dtrace_user 869

• Permanent Privilege to a User Account> Modify the file /etc/user_attr as

jack::::defaultpriv=basic,dtrace_user

> Command usermod -K defaultpriv=basic,dtrace_user jack

• Verify assigned Privileges> Command ppriv $$

Output869: bash

flags = <none>E: allI: basic,dtrace_userP: allL: all

15

Useful Links and Resources• Solaris Dynamic Tracing Guide (docs.sun.com)• BigAdmin System Administration Portal : DTrace• SDN Member Access, How To USE DTRACE from a

Solaris10 System (Excellent head start)• Advanced DTrace Tips, Tricks & Gotchas (Preso)• NetBeans or Sun Studio DTrace GUI Plugin (Refer)• DTrace Toolkit (Download)• Refer FAQ. Contact Sun PDS isvsupport@sun.com• I've started blogging at http://blogs.sun.com/stripathi

16

Exercise 1: System Calls

• Look for System Call Errors# dtrace -n 'syscall:::return /errno/ {trace(execname);trace(pid);trace(errno)}'

0 318 pollsys:return Xorg 408 4

0 12 read:return gnome-terminal 660 11

0 12 read:return Xorg 408 11

0 12 read:return nautilus 650 11

...Output truncated

• Why not use truss> truss is much easier to use, and provides better information> truss will be Invasive for Production, and may not be suitable> truss only looks at one Process> DTrace looks at a System-wide Events

Advanced than Traditional Tools

17

Exercise 2: Short-lived Malloc

• Watch for Short-lived Memory Allocations in the Application. Use ready scripts in DTraceToolkit-0.99.tar.gz# dtrace -s DTraceToolkit/Proc/shortlived.d -c firefox

dtrace: script 'shortlived.d' matched 12 probesCPU ID FUNCTION:NAME 0 1 :BEGIN Tracing... Hit Ctrl-C to stop.^C 1 2 :END short lived processes:0.052 secstotal sample duration: 11.837 secs

Total time by process name, firefox 0 ms run-mozilla.sh 1 ms pwd 2 ms awk 3 ms ls 3 ms dirname 8 ms basename 9 ms sed 13 msTotal time by PPID, 6417 0 ms 6419 0 ms[1]+ Done firefox

...Output truncated

Download the Latest DTrace Toolkit

18

Exercise 3: Who's Doing I/O

• To find out who is doing I/O on the System.Use the ready script /usr/demo/dtrace/whoio.d

# dtrace -s whoio.d

^cDEVICE APP PID BYTES

sd0 picld 168 1280

sd0 fsflush 3 3072

sd0 sched 0 295936

nfs2 sched 0 786432

sd0 soffice.bin 6070 2242048

Use Default DTrace Demo Scripts or Customize

19

Exercise 4: Syscalls by the Application

• To find which Syscalls are made by the Application.Use the ready script /usr/demo/dtrace/syscall.d# dtrace -s syscall.d -c /usr/bin/ls

dtrace: script 'syscall.d' matched 232 probes

applicat.d howlong.d pri.d spec.d whoio.dbadopen.d index.html printa.d specopen.d whopreempt.d

dtrace: pid 1053 has exitedfcntl 1fsat 1getpid 1getrlimit 1gtime 1lstat64 1rexit 1stat 1close 2fstat64 2getdents64 2mmap 2munmap 2setcontext 2ioctl 3brk 6write 21

Use Options -c or -p for PID Providers or $target Macro in Scripts

20

Exercise 5: File system Workload

• Review the script /usr/demo/dtrace/io.d# cat io.d

#!/usr/sbin/dtrace -s

#pragma D option quiet

BEGIN

{printf("%-10s %10s %10s %3s %s\n","Device", "Program","I/O Size","R/W","Path");

}

io:::start

{printf("%-10s %10s %10d %3s %s\n",args[1]->dev_statname,execname,

args[0]->b_bcount, args[0]->b_flags & B_READ? "R" : "W" ,args[2]->fi_pathname);

@[execname, pid, args[2]->fi_pathname] = sum(args[0]->b_bcount);

}

END

{

printf("%-10s %8s %10s %s\n","Program", "PID", "Total", "Path");

printa("%-10s %8d %10@d %s\n",@);

}

Use Pragma options, Macros and Functions like C Programs

21

Exercise contd: File system Workload

• Use the ready script /usr/demo/dtrace/io.d# ./io.d

On a different terminal Run mkfile 2m /demo/foo

Device Program I/O Size R/W Path

ramdisk0 mkfile 8192 R <none>

ramdisk0 mkfile 8192 W /demo/foo

ramdisk0 mkfile 8192 W /demo/foo

...Output truncated

^c

Program PID Total Path

mkfile 13262 8192 <none>

fsflush 3 25088 <none>

sched 0 33792 <none>

mkfile 13262 2105344 /demo/foo

Dynamic Tracing and Monitoring

22

Exercise 6: Process Opening Files

• Shows opened files by the Process namedtrace -n 'syscall::open*:entry { printf("%s

%s",execname,copyinstr(arg0)); }'

dtrace: description 'syscall::open*:entry ' matched 2 probes

CPU ID FUNCTION:NAME

0 2596 open:entry df /var/ld/ld.config

0 2596 open:entry df /usr/lib/libcmd.so.1

0 2596 open:entry df /usr/lib/libc.so.1

0 2596 open:entry df /etc/mnttab

^c

Command line to Enable Probes and Format Output(s)

23

Exercise 7: Disk Size by Process

• To record Actual disk I/O Requests

• Application may be doing a lot more I/O which may get absorbed by the File system cache and not result in an Actual Disk I/O

dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'

dtrace: description 'io:::start ' matched 6 probes

CPU ID FUNCTION:NAME

0 49944 bdev_strategy:start 3 fsflush 512

0 49944 bdev_strategy:start 0 sched 512

0 49944 bdev_strategy:start 0 sched 1024

Using System Probe to Trace System I/O

24

Exercise 8: Write Size by Processes

• Identify Write Size Distribution by Process on the Systemdtrace -n 'sysinfo:::writech { @dist[execname] =

quantize(arg0); }' dtrace: description 'io:::start ' matched 6 probes

^c

in.telnetd

value ------------- Distribution ------------- count

1 | 0

2 |@@@@@@@@@@@@@@@@@ 8

4 |@@ 1

8 |@@@@@@@@@@@@@ 6

16 | 0

32 | 0

64 | 0

128 | 0

256 |@@@@@@@@ 4

svc.configd

value ------------- Distribution ------------- count

...Output truncated

Use Data Manipulation and Display

25

Exercise 9: Disk Size Aggregation

• Using Aggregation for a Summarized Viewdtrace -n 'io:::start { @size[execname] =

quantize(args[0]->b_bcount); }'

dtrace: description 'io:::start ' matched 6 probes

^c

value ------------- Distribution ------------- count

512 | 0

1024 |@@ 37

2048 |@@@@@@@ 114

4096 |@@@@@@@ 116

8192 |@@@@@@@@@@@@@@@@@ 286

16384 |@@ 33

32768 |@@@@@ 87

65536 | 0

Use Data Aggregation Functions

26

Other Performance Tools

• Process Statscputrack : per-processor hardware

counterpargs : process argumentspflags : process flagspcred : process credentialspldd : process library dependencypsig : process signal dispositionpstack : process stack dumppmap : process memory mappfiles : open files and namesprstat : process statisticsptree : process treeptime : process micro-state timespwdx : process working directory

• Process Controlpgrep : grep for processespkill : kill processes listpstop : stop processesprun : start processesprctl : view/set process resourcespwait : wait for processpreap : reap a zombie process

• Process Tracing & Debugging

abitrace : trace ABI interfacesdtrace : trace the worldmdb : debug/control processestruss : trace functions,system calls

• Kernel Tracing & Debugging

dtrace : trace and monitor kernel lockstat : monitor locking statisticslockstat -k : profile kernelmdb : debug live and kernel cores

• System Statsacctcom : process accountingbusstat : Bus hardware counterscpustat : CPU hardware countersiostat : IO & NFS statisticskstat : display kernel statisticsmpstat : processor statisticsnetstat : network statisticsnfsstat : nfs server statssar : kitchen sink utilityvmstat : virtual memory stats

27

Satyajit Tripathihttp://blogs.sun.com/stripathi

27

SOLARISTM DTRACE