Hunting segfaults (for beginners)

IntroductionDetecting segfaults

Devel::Tracegdb

Devel::btThe End

Hunting segfaultsfor beginners

Uwe Volker

XING AG

August 2012

Uwe Volker Hunting segfaults


Devel::Tracegdb

Devel::btThe End

1 Introduction

2 Detecting segfaults

3 Devel::Trace

4 gdb

5 Devel::bt

6 The End



Devel::Tracegdb

Devel::btThe End

What is a segfault?Examples - CExamples - Perl

1 IntroductionWhat is a segfault?Examples - CExamples - Perl


3 Devel::Trace

4 gdb

5 Devel::bt

6 The EndUwe Volker Hunting segfaults


Devel::Tracegdb

Devel::btThe End


What is a segfault?

segfault = segmentation fault

every process has memory pages

these pages are mapped to physical memory

if you try to access an invalid address

(or write to a protected address)

BOOOM!



Devel::Tracegdb

Devel::btThe End


What is a segfault?






BOOOM!



Devel::Tracegdb

Devel::btThe End


What is a segfault?






BOOOM!



Devel::Tracegdb

Devel::btThe End


What is a segfault?






BOOOM!



Devel::Tracegdb

Devel::btThe End


Examples - C

using uninitialized pointers

dereferencing NULL pointers

using ”freed” pointers



Devel::Tracegdb

Devel::btThe End


Examples - Perl

bug in a XS extension

bug in Perl itself (rare)

Perl 5.6.1:

perl -e ’undef a’

perl -e ’*::=%::=0’

Perlmonks thread: (Golf) Segfault Perl

http://perlmonks.org/?node_id=156461




Devel::Tracegdb

Devel::btThe End


Examples - Perl



Perl 5.6.1:


perl -e ’*::=%::=0’






Devel::Tracegdb

Devel::btThe End


Examples - Perl



Perl 5.6.1:


perl -e ’*::=%::=0’






Devel::Tracegdb

Devel::btThe End

On the shellCore dump fileCGI script

1 Introduction

2 Detecting segfaultsOn the shellCore dump fileCGI script

3 Devel::Trace

4 gdb

5 Devel::bt



Devel::Tracegdb

Devel::btThe End


On the shell

p e r l s e g f a u l t . p lSegmentat ion f a u l t ( c o r e dumped )

#!/ u s r / b i n / p e r luse Debug : : DumpCore ;Debug : : DumpCore : : s e g v ;



Devel::Tracegdb

Devel::btThe End


On the shell

p e r l s e g f a u l t . p lSegmentat ion f a u l t ( c o r e dumped )

#!/ u s r / b i n / p e r luse Debug : : DumpCore ;Debug : : DumpCore : : s e g v ;



Devel::Tracegdb

Devel::btThe End


Core dump file

$ u l i m i t −c u n l i m i t e d$ p e r l s e g f a u l t . p lSegmentat ion f a u l t ( c o r e dumped )$ l l c o r e−rw−r−−−−− 1 uwe uwe 1695744 J u l 26 1 4 : 0 8 c o r e



Devel::Tracegdb

Devel::btThe End


CGI script

personal story: CGI script in Apache

no output, no entry in logfiles (access.log and error.log)

but when I wrote to some file, the content was there

so the script was getting executed...



Devel::Tracegdb

Devel::btThe End


CGI script

personal story: CGI script in Apache

no output, no entry in logfiles (access.log and error.log)

but when I wrote to some file, the content was there

so the script was getting executed...



Devel::Tracegdb

Devel::btThe End

UsageHow do I spot a segfault?Other uses for Devel::Trace

1 Introduction


3 Devel::TraceUsageHow do I spot a segfault?Other uses for Devel::Trace

4 gdb

5 Devel::bt



Devel::Tracegdb

Devel::btThe End


Usage

”Print out each line before it is executed (like sh -x)”

perl -d:Trace program

for CGI: put it in your shebang line

>> . / t e s t : 4 : p r i n t ” Statement 1 at l i n e 4\n” ;>> . / t e s t : 5 : p r i n t ” Statement 2 at l i n e 5\n” ;>> . / t e s t : 6 : p r i n t ” C a l l to sub x r e t u r n s ” , &x ( ) , ” at l i n e 6 .\ n” ;>> . / t e s t : 1 2 : p r i n t ” I n sub x at l i n e 1 2 .\ n” ;>> . / t e s t : 1 3 : return 1 3 ;>> . / t e s t : 8 : e x i t 0 ;



Devel::Tracegdb

Devel::btThe End


Usage







Devel::Tracegdb

Devel::btThe End


Usage







Devel::Tracegdb

Devel::btThe End


How do I spot a segfault?

look at the last few lines

if it stops immediately, it might be a segfault

grep for your script name

output can be very large, with long lines

grep -v site perl

in my case: buggy MSSQL driver (easysoft)



Devel::Tracegdb

Devel::btThe End







grep -v site perl




Devel::Tracegdb

Devel::btThe End







grep -v site perl




Devel::Tracegdb

Devel::btThe End


Other uses for Devel::Trace

your program is behaving strange and you have no debuggerat hand

(use grep and grep -v to filter the output)

Does this code get executed?

Which part of the conditional was taken?



Devel::Tracegdb

Devel::btThe End


Other uses for Devel::Trace

your program is behaving strange and you have no debuggerat hand

(use grep and grep -v to filter the output)

Does this code get executed?

Which part of the conditional was taken?



Devel::Tracegdb

Devel::btThe End

IntroductionUsageCore dump file - reloaded

1 Introduction


3 Devel::Trace

4 gdbIntroductionUsageCore dump file - reloaded

5 Devel::bt



Devel::Tracegdb

Devel::btThe End


Introduction

GNU debugger

command line debugger

we use it to extract the stacktrace from the core dump file



Devel::Tracegdb

Devel::btThe End


Usage

$ gdb p e r l c o r eCore was g e n e r a t e d by ‘ p e r l p e r l / s e g f a u l t . p l ’ .Program t e r m i n a t e d w i t h s i g n a l 11 , Segmentat ion f a u l t .#0 0 x00007f2 f5d086754 i n c r a s h n o w f o r r e a l ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ”) at DumpCore . xs : 1 010 p r i n t f (”%d ” , ∗p ) ; /∗ c a u s e a s e g f a u l t ∗/( gdb )



Devel::Tracegdb

Devel::btThe End


Usage

$ gdb p e r l c o r eCore was g e n e r a t e d by ‘ p e r l p e r l / s e g f a u l t . p l ’ .Program t e r m i n a t e d w i t h s i g n a l 11 , Segmentat ion f a u l t .#0 0 x00007f2 f5d086754 i n c r a s h n o w f o r r e a l ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ”) at DumpCore . xs : 1 010 p r i n t f (”%d ” , ∗p ) ; /∗ c a u s e a s e g f a u l t ∗/( gdb ) where#0 0 x00007f2 f5d086754 i n c r a s h n o w f o r r e a l ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ”) at DumpCore . xs : 1 0#1 0 x00007f2 f5d086789 i n crash now ( s u i c i d e m e s s a g e =0x 7 f 2 f 5 d 0 8 8 4 5 0 ” Cannot s t a n d t h i s l i f e anymore ” , attempt num =42) at DumpCore . xs : 1 7#2 0 x00007f2 f5d086820 i n XS Debug DumpCore segv ( cv=0x1087c10 ) at DumpCore . xs : 2 6#3 0 x0000000000488db3 i n P e r l p p e n t e r s u b ( )#4 0 x0000000000480a7d i n P e r l r u n o p s s t a n d a r d ( )#5 0 x00000000004336b4 i n p e r l r u n ( )#6 0 x000000000041bddc i n main ( )( gdb )



Devel::Tracegdb

Devel::btThe End


Core dump file - reloaded

ulimit -c unlimited

current directory has to be writable

(can be tricky with Apache)

ps auxww|grep apache

ls -l /proc/1234/cwd



Devel::Tracegdb

Devel::btThe End



ulimit -c unlimited







Devel::Tracegdb

Devel::btThe End



ulimit -c unlimited







Devel::Tracegdb

Devel::btThe End

UsageHow does it work?

1 Introduction


3 Devel::Trace

4 gdb

5 Devel::btUsageHow does it work?

6 The End



Devel::Tracegdb

Devel::btThe End


Usage

”Automatic gdb backtraces on errors”

just use the module

it registers signal handlers for SIGSEGV (and a few more)



Devel::Tracegdb

Devel::btThe End


Usage

”Automatic gdb backtraces on errors”

just use the module

it registers signal handlers for SIGSEGV (and a few more)



Devel::Tracegdb

Devel::btThe End


How does it work?

the signal handler forks off a process which runs gdb

gdb attaches to the parent and outputs the stacktrace



Devel::Tracegdb

Devel::btThe End

SourcesQuestions?

1 Introduction


3 Devel::Trace

4 gdb

5 Devel::bt

6 The EndSourcesQuestions?



Devel::Tracegdb

Devel::btThe End

SourcesQuestions?

Sources

http://en.wikipedia.org/wiki/Segmentation_fault

http://modperlbook.org/html/

21-6-Analyzing-Dumped-core-Files.html

http://www.linux-magazin.de/Heft-Abo/Ausgaben/

2007/01/Getriebeschaden


http://en.wikipedia.org/wiki/Segmentation_fault

http://modperlbook.org/html/21-6-Analyzing-Dumped-core-Files.html

http://modperlbook.org/html/21-6-Analyzing-Dumped-core-Files.html

http://www.linux-magazin.de/Heft-Abo/Ausgaben/2007/01/Getriebeschaden

http://www.linux-magazin.de/Heft-Abo/Ausgaben/2007/01/Getriebeschaden


Devel::Tracegdb

Devel::btThe End

SourcesQuestions?

Questions?


Technology

Hunting segfaults (for beginners)