Upload
pivorak-meetup
View
112
Download
1
Embed Size (px)
Citation preview
Linuxtracing
superpowersIevgen Pyrogov @gmile Lviv 2017
Plan1. BSD and
the DTrace era2. Linux and the BPF era
What is full-stack development?
BSD andthe DTrace era
A little of history
I realized at a relatively young age (~19)
that I love debugging.
Brian Cantril
The greatest satisfaction that I have had is nailing a nasty bug —
it's an experience that (for me, anyway) is so visceral as to be nearly primal.
Brian Cantril
2003
Adam Leventhal
Mike ShapiroBryan Cantrill
DTrace
# dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }' dtrace: description 'syscall::open*:entry ' matched 2 probes CPU ID FUNCTION:NAME 0 14 open:entry gnome-netstatus- /dev/kstat 0 14 open:entry man /var/ld/ld.config 0 14 open:entry man /lib/libc.so.1 0 14 open:entry man /usr/share/man/man.cf 0 14 open:entry man /usr/share/man/windex 0 14 open:entry man /usr/share/man/man1/ls.1 0 14 open:entry man /usr/share/man/man1/ls.1 0 14 open:entry man /tmp/mpqea4RF 0 14 open:entry sh /var/ld/ld.config 0 14 open:entry sh /lib/libc.so.1 0 14 open:entry neqn /var/ld/ld.config 0 14 open:entry neqn /lib/libc.so.1 0 14 open:entry neqn /usr/share/lib/pub/eqnchar 0 14 open:entry tbl /var/ld/ld.config 0 14 open:entry tbl /lib/libc.so.1 0 14 open:entry tbl /usr/share/man/man1/ls.1 0 14 open:entry nroff /var/ld/ld.config [...]
# dtrace -n 'syscall:::entry { @num[execname] = count(); }' dtrace: description 'syscall:::entry ' matched 228 probes ^C snmpd 1 utmpd 2 inetd 2 nscd 7 svc.startd 11 sendmail 31 poold 133 dtrace 1720
# dtrace -n 'syscall:::entry { @num[probefunc] = count(); }' dtrace: description 'syscall:::entry ' matched 228 probes ^C fstat 1 setcontext 1 lwp_park 1 schedctl 1 mmap 1 sigaction 2 pset 2 lwp_sigmask 2 gtime 3 sysconfig 3 write 4 brk 6 pollsys 7 p_online 558 ioctl 579
$ dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }' dtrace: description 'syscall:::entry ' matched 228 probes ^C 1109 svc.startd 1 4588 svc.startd 2 7 svc.startd 2 3950 svc.startd 2 1626 nscd 2 870 svc.startd 2 82 nscd 6 5011 sendmail 10 6010 poold 74 8707 dtrace 1720
$ dtrace -n 'sysinfo:::readch { @bytes[execname] = sum(arg0); }' dtrace: description 'sysinfo:::readch ' matched 4 probes ^C mozilla-bin 16 gnome-smproxy 64 metacity 64 dsdm 64 wnck-applet 64 xscreensaver 96 gnome-terminal 900 ttymon 5952 Xorg 17544
$ dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }' dtrace: description 'proc:::exec-success ' matched 1 probe CPU ID FUNCTION:NAME 0 3297 exec_common:exec-success man ls
0 3297 exec_common:exec-success tbl /usr/share/man/man1/ls.1 0 3297 exec_common:exec-success neqn /usr/share/lib/pub/eqnchar - 0 3297 exec_common:exec-success nroff -u0 -Tlp -man - 0 3297 exec_common:exec-success col -x 0 3297 exec_common:exec-success sh -c more -s /tmp/mpzIaOZF 0 3297 exec_common:exec-success more -s /tmp/mpzIaOZF
dtrace -l dtrace: system integrity protection is on, some features will not be available dtrace: failed to initialize dtrace: DTrace requires additional privileges
Brendan Gregg
One the first guys to use DTrace
outside of SUN
January 27, 2010
ResignationsPetitions and forks
LawsuitResignations
Forks
Forks
Program closures
Linux andthe BPF era
Linuxtracingtools
SystemTap trace strace
sysdig ktap
LTTng
Limitations:
1. Expensive to run
2. Extremely hard to use in production
3. May be unsafe to run
Result: no single consolidated effort in
development
BPF
Berkeley Packet Filter
BPF – is a tiny VM inside the kernel
Initial purpose – to filter TCP packets!
How BPF looked like back then:
l0: ldh [12] l1: jeq #0x800, l3, l2 l2: jeq #0x805, l3, l8 l3: ld [26] l4: jeq #SRC, l4, l8 l5: ld len l6: jlt 0x400, l7, l8 l7: ret #0xffff l8: ret #0
2013
Things startto change
dramatically
Alexei Starovoitov
BPF is brutally hard to write!
Far-far from DTrace one-liners
Brendan Gregg: “BPF is the only language that defeated me!”
We should not write BPF programs
directly
Enter BCC!
BCC
BCC = BPF Compiler
Collection
1. Write BPF program in restricted C
2. Let Python or Lua leverage the rest
1. Write your BPF program in C... inline or in a separate file
2. Write a python script that loads and interacts with your BPF program
3. Attach to kprobes, socket, etc.
4. Read/update maps
5. Configuration, complex calculation/correlations
6. Iterate on above and re-try...in seconds
from bpf import BPF from subprocess import call
prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }; “""
b = BPF(text=prog)
fn = b.load_func("hello", BPF.KPROBE)
BPF.attach_kprobe(fn, “sys_clone")
try: call(["cat", "/sys/kernel/debug/tracing/trace_pipe"]) except KeyboardInterrupt: pass
[root@localhost examples]# ./hello_world.py python-20662 [001] d..1 1138.551706: : Hello, World! tmux-1012 [002] d..1 1139.227627: : Hello, World! tmux-1012 [002] d..1 1139.229636: : Hello, World! byobu-20664 [006] d..1 1139.235396: : Hello, World! byobu-20665 [007] d..1 1139.236660: : Hello, World! byobu-20665 [007] d..1 1139.246109: : Hello, World! ^C
Brenden Blanco
Largest difference between
DTrace and BPF is: ease of use
Simpler tools utilizing BPF are
already being built!
1. BPF backend for perf command
2. ply
#!/usr/bin/env ply
kprobe:SyS_* { $syscalls[func].count() }
3. BPF backend for SystemTap
#!/usr/bin/stap /* * opensnoop.stp Trace file open()s. Basic version of opensnoop. */
probe begin { printf("\n%6s %6s %16s %s\n", "UID", "PID", "COMM", "PATH"); }
probe syscall.open { printf("%6d %6d %16s %s\n", uid(), pid(), execname(), filename); }
Find / fix bcc issues
High-level language
Promotion
Promotion
Education
GUI integration
Examples
# ./execsnoop PCOMM PID RET ARGS bash 15887 0 /usr/bin/man ls preconv 15894 0 /usr/bin/preconv -e UTF-8 man 15896 0 /usr/bin/tbl man 15897 0 /usr/bin/nroff -mandoc -rLL=169n -rLT=169n -Tutf8 man 15898 0 /usr/bin/pager -s nroff 15900 0 /usr/bin/locale charmap nroff 15901 0 /usr/bin/groff -mtty-char -Tutf8 -mandoc -rLL=169n -rLT=169n groff 15902 0 /usr/bin/troff -mtty-char -mandoc -rLL=169n -rLT=169n -Tutf8 groff 15903 0 /usr/bin/grotty
# ./opensnoop PID COMM FD ERR PATH 17326 <...> 7 0 /sys/kernel/debug/tracing/trace_pipe 1576 snmpd 9 0 /proc/net/dev 1576 snmpd 11 0 /proc/net/if_inet6 1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/eth0/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/conf/eth0/forwarding 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/eth0/base_reachable_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv4/neigh/lo/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/retrans_time_ms 1576 snmpd 11 0 /proc/sys/net/ipv6/conf/lo/forwarding 1576 snmpd 11 0 /proc/sys/net/ipv6/neigh/lo/base_reachable_time_ms 1576 snmpd 9 0 /proc/diskstats 1576 snmpd 9 0 /proc/stat 1576 snmpd 9 0 /proc/vmstat 1956 supervise 9 0 supervise/status.new 1956 supervise 9 0 supervise/status.new 17358 run 3 0 /etc/ld.so.cache 17358 run 3 0 /lib/x86_64-linux-gnu/libtinfo.so.5 17358 run 3 0 /lib/x86_64-linux-gnu/libdl.so.2 17358 run 3 0 /lib/x86_64-linux-gnu/libc.so.6 17358 run -1 6 /dev/tty 17358 run 3 0 /proc/meminfo 17358 run 3 0 /etc/nsswitch.conf 17358 run 3 0 /etc/ld.so.cache 17358 run 3 0 /lib/x86_64-linux-gnu/libnss_compat.so.2 17358 run 3 0 /lib/x86_64-linux-gnu/libnsl.so.1 17358 run 3 0 /etc/ld.so.cache 17358 run 3 0 /lib/x86_64-linux-gnu/libnss_nis.so.2 17358 run 3 0 /lib/x86_64-linux-gnu/libnss_files.so.2 17358 run 3 0 /etc/passwd 17358 run 3 0 ./run
# ./bashreadline TIME PID COMMAND 05:28:25 21176 ls -l 05:28:28 21176 date 05:28:35 21176 echo hello world 05:28:43 21176 foo this command failed 05:28:45 21176 df -h 05:29:04 3059 echo another shell 05:29:13 21176 echo first shell again
# ./funccount 'vfs_*' Tracing... Ctrl-C to end. ^C FUNC COUNT vfs_create 1 vfs_rename 1 vfs_fsync_range 2 vfs_lock_file 30 vfs_fstatat 152 vfs_fstat 154 vfs_write 166 vfs_getattr_nosec 262 vfs_getattr 262 vfs_open 264 vfs_read 470 Detaching...
# trace 'sys_execve "%s", arg1' PID COMM FUNC - 4402 bash sys_execve /usr/bin/man 4411 man sys_execve /usr/local/bin/less 4411 man sys_execve /usr/bin/less 4410 man sys_execve /usr/local/bin/nroff 4410 man sys_execve /usr/bin/nroff 4409 man sys_execve /usr/local/bin/tbl 4409 man sys_execve /usr/bin/tbl 4408 man sys_execve /usr/local/bin/preconv 4408 man sys_execve /usr/bin/preconv 4415 nroff sys_execve /usr/bin/locale 4416 nroff sys_execve /usr/bin/groff 4418 groff sys_execve /usr/bin/grotty 4417 groff sys_execve /usr/bin/troff
# trace 'sys_read (arg3 > 20000) "read %d bytes", arg3' PID COMM FUNC - 4490 dd sys_read read 1048576 bytes 4490 dd sys_read read 1048576 bytes 4490 dd sys_read read 1048576 bytes 4490 dd sys_read read 1048576 bytes
trace 'r:bash:readline "%s", retval' PID COMM FUNC - 2740 bash readline echo hi! 2740 bash readline man ls
Bonus track
Thanks!
Questions?