Debugging Techniques

AdvancedSoftwareDevelopmentEngineering

1

Paolo AdragnaUniversit degli Studi di Siena

Debugging Techniques


2

WhyDebugging?

Debuggingisafundamentalpartofprogrammers'everydayactivity....

...butsomepeopleconsideritanannoyingoption...


3

USSYorktown(1998)

AcrewmemberoftheguidedmissilecruiserUSSYorktownmistakenlyenteredazeroforadatavalue,whichresultedinadivisionbyzero.Theerrorcascadedandeventuallyshutdowntheship'spropulsionsystem.Theshipwasdeadinthewaterforseveralhoursbecauseaprogramdidn'tcheckforvalidinput.(reportedinScientificAmerican,November1998)


4

MarsClimateOrbiter(1999)Theprocessdidnotspecifythesystemofmeasurementtobeusedontheproject.Asaresult,oneofthedevelopmentteamsusedImperialmeasurementwhiletheotherusedthemetricsystem.Whenparametersfromonemodulewerepassedtoanotherduringorbitnavigationcorrection,noconversionwasperformed,resultinginthelossofthecraft.http://mars.jpl.nasa.gov/msp98/orbiter/

The125milliondollarMarsClimateOrbiterisassumedlostbyofficialsatNASA.ThefailureresponsibleforlossoftheorbiterisattributedtoafailureofNASAssystemengineerprocess.


5

LectureProgramme

PartIGeneralAspectsofDebugging

PartIIGeneralDebugging

PartIIIC/C++RelatedProblemsandSolvers


6

PartI

GeneralAspectsofDebugging


7

PartOneGeneralAspectsofDebugging

Thedebuggingprocessinvolves:

Localisingabug Classifyingabug Understandingabug Repairingabug


8

LocalisingaBug

YouknowwhatyourcodeshoulddoYounoticeitdoesnotdothatsonoticingabugiseasy,youmightsay...

#include//Ascopingexamplevoidc(void);//functionprototype

intx=1;//globalvariable

intmain(){intx=5;//localtomain//Someothercodewhile(x


9

ClassifyingaBug

Sinceexperienceswithbugshaveoftenacommonbackground,wemayattemptaclassification: SyntacticalErrors:errorsyourcompilershouldcatch.

BuildErrors:errorsfromusingobjectfilesnotrebuiltafterachangeinsomesource.

BasicSemanticErrors:usinguninitializedvariables,deadcode,typeproblems.

SemanticErrors:usingwrongvariables,exchangingoperator(e.g.&insteadof&&)


10

ClassifyingaBug

AfunnyphysicalclassificationBohrbugsandHeisembugs

Bohrbugsaredeterministic:aparticularinputwillalwaysmanifestthem.

Heisembugsarerandom:difficulttoreproducereliably


11

UnderstandingaBug

Understandabugfullybeforeattemptingtofixit

Askyourselfsomequestions: HaveIfoundthesourceoftheproblemoronlyasymptom?

HaveImadesimilarmistakes(especiallywrongassumptions)elsewhereinthecode?

Isthisonlyaprogrammingerrororisthereamorefundamentalproblem(e.g.incorrectalgorithm)?


12

RepairingaBug

Repairingabugismorethanmodifyingcode.Makesureyoudocumentyourfixinthecodeandtestitproperly.

Afterrepair,whatdidyoulearnfromit? Howdidyounoticethebug?Thismayhelpyouwritingatestcase.

Howdidyoutrackitdown?Thiswillgiveyouabetterinsightontheapproachtochooseinsimilarcircumstances.

Whattypeofbugdidyouencounter?


13

RepairingaBug

Afterrepair,whatdidyoulearnfromit? Doyouencounterthisbugoften?Ifso,whatcouldyoudotopreventitfromreoccurring?

Whatyouhavelearntisvaluable:trytocommunicateitwithyourcollegues

Unjustifiedassumptions?

Afterrepairingabug,writeatestcasetomakesureitdoesnothappenagain


14

PartTwo

GeneralDebugging


15

PartTwoGeneralDebugging

A)ExploitingCompilerFeature

B)ReadingTheRightDocumentation

C)TheAbusedcoutDebuggingTechnique

D)Logging

E)DefensiveProgramming

F)ACIDebuggingTechnique

G)WalkingThroughTheCode

H)TheDebugger


16

ExploitingCompilerFeatures(General)

Agoodcompilercandoanamountofstaticanalysisonyourcode(theanalysisofthoseaspectsthatcanbestudiedwithoutexecution)

Staticanalysiscanhelpindetectinganumberofbasicsemanticproblems(e.g.typemismatch,deadcode)


17

ExploitingCompilerFeatures(gcc)

Forgccthereareanumberofoptionsthataffectwhichstaticanalysiscanbeperformed WallW

Alsorecommendedwhenwritingnewcode Wshadow Wpointerarith Wcastequal Wcastalign Wstrictprototype


18

ExploitingCompilerFeature(gcc)

Anumberofoptimizationsaresupported.Someofthesetriggergcctodoextensivecodeflowanalysis,removingdeadcode.

Recommendedfornormaluse:O2 Warning:optimisationkillsdebugging,soyouhavetochoose Example:gccO3orgccgO0


19

ReadingtheRightDocumentation

Takethetimetofindatyourfingertipsrelevantdocumentationfor: yourtask yourtools yourlibraries

youralgorithm

Youdonotneedtoknoweverything Youneedtobeawarewhatdocumentationisrelevantandwhatisitspurpose


20

TheAbusedcoutTechnique

Thistechniqueisencounteredtoooften. Itconsistsofadhocinsertionoflotofprintingstatementtotrackthecontrolflowanddatavaluesduringtheexecutionofapieceofcode

Disadvantages Itisveryadhoc Itclobbersthenormaloutput

Slowstheprogramdownconsiderably Oftenitdoesnothelp(outputbuffered)


21

TheAbusedcoutTechnique

Ifyouconsiderusingdebugging,checkouttheuseofassertionandofadebugger,muchmoreeffectiveandtimesaving

Insomecircumstancescoutdebuggingisappropriate.Sometips: Produceoutputonstandarderror(unbuffered) Donotuseprintingstatementsdirectly:defineamacroaroundthem

Usedebuggingleveltomanagetheamountofdebugginginformation


22

coutTechniqueExample

#ifndefDEBUG_H#defineDEBUG_H#include

#ifdefined(NDEBUG)&&defined(__GNUC__)/*gcc'scpphasextensions;itallowsformacroswithavariablenumberofarguments.Weusethisextensionheretopreprocesspmesgaway.*/#definepmesg(level,format,args...)((void)0)#elsevoidpmesg(intlevel,char*format,...);/*printamessage,ifitisconsideredsignificantenoughAdaptedfrom[K&R2],p.174*/#endif#endif/*DEBUG_H*/


23

Logging

Loggingisacommonaidtodebugging Heavilyusedbydaemonandservices Itisarealsolutiontothecouttechnique Itrecordsinformationmessageswhichmonitorthestatusofyourprogram

Theycanevenformthebasisofsoftwareauditing

Asensiblemethodistoclassifylogmessagesandlabelthemwithaprioritylevel


24

log4cppC++Logging

Log4cpphas3maincomponents: Categories Appenders Layouts

Alayoutclasscontrolswhattheoutputmessageisgoingtolooklike.

YoumayderiveyourownclassesfromLayoutorusetheprovidedSimpleLayoutandBasicLayout


25

log4cppC++Logging

Anappenderclasswritesthetracemessage,formattedbyalayoutobject,outtosomedevice

log4cppcomeswithclassestoappendtostandardoutput,anamedfile,orastringbuffer:FileAppenderOstreamAppenderStringQueueAppender

Onceagainyoumayderiveyourownappender(e.g.toasocket,asharedmemorybuffer...)


26

log4cppC++Logging

Acategoryclassdoestheactuallogging.

Thetwomainpartsofacategoryareitsappendersanditspriority

Thepriorityofacategorycanbesetto:

1NOTSET

2DEBUG

3INFO

4NOTICE

5WARN

6ERROR

7CRIT

8ALERT

9FATAL/EMERGinascendingorderofimportancelevel


27

log4cppC++Logging

Eachmessageisloggedtoacategoryobject

Thecategoryobjecthasaprioritylevel

Prioritycontrolswhichmessagescanbeloggedbyaparticularclass.

Themessageitselfalsohasaprioritylevelasitwendsitswaytothelog

Ifthepriorityofthemessageisgreaterthan,orequalto,thepriorityofthecategory,thenloggingtakesplace,otherwisethemessageisignored


28

Log4cppExample

Instantiateanappenderobjectthatwillappendtoalogfile

log4cpp::Appender*app=newlog4cpp::FileAppender("FileAppender","/logs/testlog4cpp.log");

Instantiatealayoutobject

log4cpp::Layout*layout=newlog4cpp::BasicLayout();

Attachthelayoutobjecttotheappender

app>setLayout(layout);

Therearesixinitialstepstousingalog4cpplog:


29

Log4cppExample

Instantiateacategoryobjectbycallingthestaticfunction

log4cpp::Layout*layout=newlog4cpp::BasicLayout();

Attachtheappenderobjecttothecategoryasanadditionalappender(inadditiontothedefaultstandardoutappender),orsetAdditivitytofalsefirstandinstalltheappenderastheoneandonlyappenderforthatcategory

main_cat.setAppender(app);

Setapriorityforthecategory

main_cat.setPriority(log4cpp::Priority::INFO);


30

Log4cppExample

Someexamples:main_cat.info("Thisissomeinfo");main_cat.debug("Thisdebugmessagewillfailtowrite");main_cat.alert("Allhandsabandonship");

/*youcanlogbyusingalog()methodwithapriority*/main_cat.log(log4cpp::Priority::WARN,"Thiswillbealoggedwarning");

/*thiswouldnotbeloggedifpriority==DEBUG,becausethecategorypriorityissettoINFO*/main_cat.log(priority,"Importancedependsoncontext");

Otherexampleinthecitedpaper(seeBibliography)


31

Log4cppLogfileExample

Atipicallogfile:

995871335INFOmain_cat:Thisissomeinfo995871335PANICmain_cat:Allhandsabandonship995871335WARNmain_cat:Thiswillbealoggedwarning995871335ALERTmain_cat:Importancedependsoncontext995871335ERRORmain_cat:Andthiswillbeanerror995871335INFOmain_cat:info995871335NOTICEmain_cat:notice995871335WARNmain_cat:warn


32

DefensiveProgrammingandtheassertMacro Takealookatyourcode:ineverypartyoumakealotofassumptionsaboutotherparts

Assertionsareexpressionsyoushouldevaluatetobetrueataspecificpointinyourcode

Ifanassertionfails,youhavefoundaproblem(possiblyintheassertion,morelikelyinthecode)

Itmakenosensetoexecuteafteranassertionfails


33

DefensiveProgrammingandtheassertMacro

Writingassertionsmakesyourassumptionsexplicit

InC/C++youcan#includeandwritetheexpressionyouwanttoassertasmacroargument

Withassertmacrosyourprogramwillbeabortedwhenanassertionfails

Anassertionfailureisreportedbyamessage


34

ACIDebuggingTechnique

ACI,onlyajoke... ThetechniquenamederivefromAutomobileClubd'Italia,anItalianorganisationthathelpswithcartroubles...


35

ACIDebuggingTechnique

ACI,notonlyajoke... Basedonasimpleprinciple:thebestwaytolearnthingistoteachthem

InACIdebuggingyoufindabystanderandexplaintoherhowyourcodeworks

ThisforcesyoutorethinkyourassumptionandexplainwhatisreallyhappeningItcanbeaformofpeerreview


36

WalkingthroughtheCode

ThistechniqueissimilartotheACItechnique.

Therecipe: Printyourcode Leaveyourterminal Gotocafeteria Takethebeverageofyourchoice,ifpossiblewithcaffeineandsugar

Readyourcodeandannotateitcarefully


37

TheDebugger

Wheneveryothercheckingtoolfailsdetectingtheproblem,thenitisdebugger'sturn.

Adebuggerallowstoworkthroughthecodelinebylinetofindoutwhereandwhyitisgoingwrong.

Youcaninteractivelycontroltheprogramrun,stopitatvarioustimes,inspectvariables,changecodeflowwhilstrunning.


38

TheDebugger

Inordertomakeuseofadebugger,aprogrammustbecompiledwithdebugginginformationinserted(debuggingsymbols)

Debuggingsymbolsdescribewherethefunctionandvariablesarestoredinmemory

Anexecutableswithdebuggingsymbolscanrunasanormalprogram,evenifslightlyslower


39

Breakpoints

Breakpointsstopaprogramwhenneeded Theprogramrunsnormallyuntilitisabouttoexecutethepieceofcodeatthesameaddressofthebreakpoint

atthatpoint,theprogramdropsbackintothedebuggerandwecanlookatvariables,orcontinuesteppingthroughthecode.

Breakpointsarefundamentalininteractivedebugging


40

Breakpoints

Breakpointshavemanyoptions.Theycanbesetup: onaspecificlinenumber atthebeginningofafunction ataspecificaddress

conditionally


41

DebuggingCommands

Afterstopping(e.g.atabreakpoint)everydebuggercan:

executenextprogramlinesteppingoveranyfunctioncallsintheline

executenextprogramlinesteppingintoanyfunctioncallsintheline

continuingrunningyourprogram


42

Watchpoints

Watchpointsareaparticulartypeofbreakpoints

Awatchpointstopsthecodewheneveravariablechanges,evenifthelinedoesn'treferencethevariableexplicitlybyname

Awatchpointlooksatthememoryaddressofthevariableandalertsyouwhensomethingiswrittentoit


43

BinarySplit

Inlargeprograms,addingbreakpointsforeveryiterationoftheloopisprohibitive

Itisnotnecessarytostepthrougheachoneinturn,butemployatechniqueknownasbinarysplit: Weplaceabreakpointafterthefirstofthecodeandrunit.

Iftheproblemhasnotshowedup,thenitislikelytobeafaultwiththelasthalf.


44

BinarySplit

Fromhere,wecanaskthequestionagain,reducingtheareaundertesttothefirstorthesecondquarter

Thisquestioncanbeaskedrepeatedlyuntilwe'redowntojustoneline,orsufficientlysmallroutinethatwecanstepthroughlinebyline

Abinarysplitcanlimitthesearchareaofa1000lineprogramtojust10steps!


45

DDDGUIforgdb

Tryitonourfirstexample

DataDisplayDebugger

Powerfulinterfacetogdbwithextrafeatures


46

PartIII

C/C++RelatedProblemsandSolvers


47

PartThreeC/C++RelatedProblemsandSolvers

A)Preprocessor

B)DynamicStorageAllocation

C)SystemCallExamination


48

C/C++BuildProcess

Abriefreviewofstepsinvolvedinbuildingandrunningaprogram

Preprocessingheaderfiles,inclusionandmacroprocessing;outputinpureC/C++code

CompilingtranslationofpureC/C++codetoassemblylanguage

Assemblingtranslationofassemblycodeintobinaryobjectcode


49

C/C++BuildProcess

Linkinglinkercombinesanumberofobjectfilesandlibrariestoproduceexecutablesorlibraries

DynamicLoadinglibraries(orlibraryparts)requiredbyadynamicallylinkedexecutablesareloadedpriortoactualrunningtheexecutables


50

Preprocessor

TheC/C++preprocessor: expandsmacros

declaresdependencies drivesconditionalcompilation

Preprocessoroperationsareperformedattextuallevel.Thiscanmaketrackingdownmissingdeclarationdifficultorleadtosemanticproblem


51

Preprocessor

Ifyoususpectapreprocessingproblem,letthepreproccessorexpandthefileforexamination

Example:gccE Stopsafterthepreprocessingstagewithoutrunningthecompiler.Theoutputispreprocessedsourcecode,whichissenttothestandardoutput


52

DynamicStorageAllocation

InC/C++youhavetoexplicitlyallocateanddeallocatedynamicstorage(throughmalloc/freeornew/delete).

Ifmemoryis(de)allocatedincorrectly,itcancauseproblemsatruntime(e.g.memorycorruption,memoryleak)

Commonerrorsare:tryingtousememorythathasnotbeenallocatedyet;toaccessmemoryalreadydeallocated;deallocatingmemorytwice


53

MemoryAllocationDebuggingTools

Whenyouhaveamemoryproblem,thebestitcanhappenisaprogramcrash!!!

Basicallytwocategoriesoftools: Externallibrariestobeincludedand/orlinked

MEMWATCH ElectricFence

Executableswhichcontrolsprogram'srun YAMD Valgrind


54

ElectricFence

ElectricFenceisClibraryformallocdebugging Itexploitsthevirtualmemoryhardwareofthesystemtocheckifandwhenaprogramexceedsthebordersofamallocbuffer.

Atthebordersofsuchbuffer,aredzoneisadded.Whentheprogramentersthiszone,itisterminatedimmediately.

Thelibrarycanalsodetectwhentheprogramtriestoaccessmemoryalreadyreleased.


55

ElectricFence

BecauseElectricFenceusestheVirtualMemoryhardwaretodetecterrors,theprogramwillbestoppedatthefirstinstructionthatcausesacertainbuffertobeexceeded.

Thereforeitbecomestrivialtoidentifytheinstructionthatcausedtheerrorwithadebugger

Whenmemoryerrorsarefixed,itisbettertorecompiletheprogramwithoutthelibrary.


56

ExampleMemoryError

intmain(intargc,char*argv[]){double*histo;histo=(double*)malloc(sizeof(double)*60));for(inti=0;i


57

Valgrind

Valgrindcheckseveryreadingandwritingoperationonmemory,interceptingallcallstomalloc/freenew/delete

Valgrindcandetectproblemslike: usageofuninitialisedmemory readingfrom/writingtofreedmemory readingfrom/writingbeyondthebordersofallocatedblocks


58

Valgrind

Valgrindtrackseverybyteofthememorywithninestatusbits:onefortheaccessibilityandtheothereightforthecontent,ifvalid.

Asaconsequence,Valgrindcandetectuninitialisedanddoesnotreportfalseerrorsonbitfieldoperations.

ValgrindcandebugalmostalldynamicallylinkedELFx86executableswithoutanyneedformodificationorrecompilation.


59

ExampleMemoryError

intmain(intargc,char*argv[]){double*histo=newdouble[60];for(inti=0;i


60

ExampleMemoryError

valgrindgdbattach=yeserrorlimit=no./memerror.....==3252==Invalidwriteofsize8==3252==at0x80483DA:main(memerror.cpp:9)==3252==by0x4026F9B1:__libc_start_main(in/lib/libc.so.6)==3252==by0x80482F0:???(start.S:102)==3252==Address0x410B2204is0bytesafterablockofsize480alloc'd==3252==at0x4002ACB4:malloc(in/usr/lib/valgrind/vgskin_memcheck.so)==3252==by0x80483A8:main(memerror.cpp:7)==3252==by0x4026F9B1:__libc_start_main(in/lib/libc.so.6)==3252==by0x80482F0:???(start.S:102)==3252====3252==AttachtoGDB?[Return/N/n/Y/y/C/c]


61

ExampleForgettingtheInitialisationConsiderthefollowingsimpleprogram

#includeintmain(intargc,char*argv[]){doublek,l;doubleinterval=atof(argv[1]);if(interval==0.1){k=3.14;}if(interval==0.2){k=2.71;}l=5.0*exp(k);std::cout


62

ExampleForgettingtheInitialisationvalgrindgdbattach=yeserrorlimit=noleakcheck=yesmemerror

==3252==Invalidwriteofsize8==3252==at0x80483DA:main(memerror.cpp:9)==3252==by0x4026F9B1:__libc_start_main(in/lib/libc.so.6)==3252==by0x80482F0:???(start.S:102)==3252==Address0x410B2204is0bytesafterablockofsize480alloc'd==3252==at0x4002ACB4:malloc(in/usr/lib/valgrind/vgskin_memcheck.so)==3252==by0x80483A8:main(memerror.cpp:7)==3252==by0x4026F9B1:__libc_start_main(in/lib/libc.so.6)==3252==by0x80482F0:???(start.S:102)==3252==AttachtoGDB?[Return/N/n/Y/y/C/c]


63

ExampleTrackingMemoryLeak#includeusingnamespacestd;string&xform_string_copy(conststring&input);

intmain(intargc,char*argv[]){std::stringoriginal("Iamanautomaticvariable");string&stringref=xform_string_copy(original);}string&xform_string_copy(conststring&input){string*xformed_p=newstring("Iwillprobablybeleaked!");//...maybedosomeprocessinghere...return*xformed_p;//Callerswillalmostneverfreethisobject.}

TypicalError

ReturningaReferencetoaDynamicallyAllocatedObject


64

SystemCallExamination

ASystemCallTracerallowsyoutoexamineproblemsattheboundarybetweenyourcodeandoperatingsystem

Thetracershowswhatsystemcallsaprocessmakes(withparametersandreturnvalue)

Atracercannottellyouwhereasystemcallwasmadeinyourcode.

Theexactplacehastobereconstructed


65

strace,theLinuxSystemTracer

straceisapowerfultoolwhichshowsallthesystemcallsissuedbyauserspaceprogram.

stracedisplaystheargumentstothecallsandreturnsvaluesinsymbolicform.

stracereceivesinformationfromthekernelanddoesnotrequirethekerneltobebuiltinanyspecialway.


66

straceexample

#include//forI/O#include//forstrings#include//forfileI/O#include//forexit()

usingnamespacestd;

intmain(intargc,char*argv[]){stringfilename;stringbasename;stringextname;stringtmpname;conststringsuffix("tmp");


67

straceexample

/*foreachcommandlineargument(whichisanordinaryCstring)*/for(inti=1;i


68

straceexample

ifstreamfile(tmpname.c_str());if(!file){cerr


69

straceexample

...butthereitis!


70

straceexample

Let'sstartstrace:straceostrace.outstracexlist

brk(0x804a76c) =0x804a76cbrk(0x804b000)=0x804b000open("list",O_RDONLY)=1ENOENT(Nosuchfileordirectory)write(2,"C",1) =1write(2,"a",1)=1write(2,"n",1) =1write(2,"\'",1) =1write(2,"t",1)=1write(2,"",1)=1write(2,"o",1)=1write(2,"p",1)=1write(2,"e",1)=1write(2,"n",1)=1


71

Acknowledgments

IwouldliketothankverymuchJ.H.M.DassenandI.G.SprinkhuizenKuyperforlettingmeusesomeoftheirmaterialondebuggingtechniques

AbigthankalsotoP.F.Zema,mycollegueinATLAS,forusefultechnicalcommentsandideasexchangeonLinuxdebugging.

ThankstoE.Castorinaforacriticalreviewofthelectureslides


72

Bibliography

Formorefamousbugs,takealooktoProf.GSantor'ssite:http://infotech.fanshawec.on.ca/gsantor/Computing/FamousBugs.htm

J.H.M.Dassen,I.G.SprinkhuizenKuyper,DebuggingCandC++codeinaUnixenvironment,UniversiteitLeiden,Leiden,1999

T.Parr,Learntheessentialofdebugging,IBMdeveloperWorksjournal,Dec2004

S.Best,MasteringLinuxdebuggingtechniques,IBMdeveloperWorksjournal,Aug2002

S.Goodwin,ThePleasurePrinciple,LinuxMagazine31(2003)6469


73

Bibliography

gdbUserManual

gccUserManual

ValgrindUserManual

F.Rooms,SomeadvancedtechniquesinCunderLinux

W.Mauerer,VisualDebuggingwithddd,TheLinuxGazette,Jan2001

M.Budlong,LoggingandTracinginC++Simplified,SunDevelopersTechnicalArticles,2001

S.Goodwin,D.Wilson,WalkingUpright,LinuxMagazine27(2003)7680

J.World,UsingLog4c,onlineathttp://jefficus.usask.ca


74

BackupSlides


75

LocalisingaBug

Youknowwhatyourcodeshoulddo,younoticeitdoesnotdothatsonoticingabugiseasy,youmightsay...

Noticingabugimpliestesting,sothiseasinessiscompletelydeceptive

Incaseofatestfailureyouhavetoseewhatwentwrong,soprepareyourtestscarefully


76

Introduction

Whenyourprogramcontainsabug,itislikelythat,somewhereinthecode,aconditionyoubelievetobetrueisactuallyfalse

Findingyourbugisaprocessofconfirmingwhatyoubelieveistrueuntilyoufindsomethingthatisfalse.

Myprogramdoesn'tworkisnotanacceptablestatement


77

Introduction

Theimportanceofthewayhowtofinderrorsandfixtheminthelifecycleofasoftwareproductisataskwhoseimportancecannotbestressedenoughoverandover

Findingerrorsisnotjustanunavoidablepartinthedevelopmentcyclebutvitalpartofeverysoftwaresystem'slifespan.

Documents

Debugging Techniques