Troubleshooting Linux

Keith WrightTechnical Training Resources, Inc.

Troubleshooting

• Be prepared!• What's the problem?• What does the problem affect?• What or who is to blame?• Searching for solutions• Solving the problem or not!• What's next?

Be Prepared

• Documentation• Back it up!

o Software Back up any important data Have a regular schedule Automate Backup configuration changes Prepare rescue media

o Hardware Have spare parts available Have spare machines Use Virtualization/Clustering

What's the problem?

• Gather Informationo Usero gnome-screenshot –interactiveo Run from terminalo Log files (/var/log)o /usr/bin/gnome-system-logo historyo strace/ltraceo top, ps, sar, iostat, vmstato tcpdump, wireshark, ntop, snorto Debugging Options

Network

Version

Architecture

Machine

Everyone

How widespread is problem?• Research • Try other systems• If identical systems work

o diff configuration fileso Use alternatives

• Other versions• Other software• Other architectures• Build from source• Submit/Fix a bug Software

What are likely causes?

• User error• Software configuration changes• Missing dependencies• Updates to software• Updates to hardware• Hardware problems • Updates to network• Network problems

Quick Fixes

• Software • Reboot• Reinstall software• add-apt-repository (/etc/apt/sources.list)• Update (apt-get update;apt-get upgrade –f)

• Hardware -> • System, Administration, Hardware Drivers• System, Administration, System Testing• Shutdown and Boot• Check cables, connections, indicator lights, and

power• Ubuntu: Recovery Mode

• Clean, dpkg, grub, netroot, root

Troubleshooting Boot

• grub legacy: /boot/grub/menu.lst• grub 2: /boot/grub/grub.cfg• startupmanager package• Recovery Mode• Append kernel arguments

o 1 (Single user mode)o emergency (Just a shell)

• linux rescue (Get rescue distro CD or...)o package: grub-rescue-pc o mount /dev/rootpartition /targeto dpkg --root /target --install package.deb

• sysv-rc-conf or rcconf to manage daemons • /etc/default/* for startup parameters• /etc/init.d/* for daemon scripts• /etc/init/* for system initialization scripts

Troubleshooting X11

• gnome-display-properties• ~/.config/monitors.xml• ~/.gconfd/saved_state • ~/Xclients-default• /etc/X11/xorg.conf• /var/log/Xorg.0.log• CTRL+ALT+BACKSPACE• ATI & Nvidia drivers

• System, Administration, Hardware Drivers• Need to build kernel module• build-essential• linux-headers-$(uname –r)

GNOME Desktop Problems

• gconf-editor• /apps/panel/global/locked_down• /desktop/gnome/lockdown • rm -rf ~/.gconf ~/.gconfd ~/.gnome2

~/.gnome2_private

Troubleshooting Processes

• ps -ef | grep foo• pkill (-9) foo, pkill -u user• kill -9 pid• xkill• top, htop, atop, gnome-system-monitor• latencytop, powertop

GUI Applications• Is a dialog box displayed?

o Press the PrintScr key on the keyboard and save the message

o Save the file for reference or being able to share it with others

• Run Application from Terminalo Right-click the menu item, select Add this laucher to

the desktopo Right-click the item on the desktop, select Properties,

and copy the commando Open a Terminal and execute the commando Look for informative error messages

Application crashes

• Open a terminal• Read the manual and start the Application in a

debugging mode• Perform operation that caused crash• Analyze any error messages• Save output• Research error message

Command Not Found or not working as expected

• Was the command typed correctly?• Is the file executable to the user• Is the file located on the user's PATH?

o Echo $PATHo Export PATH=/foo/bar:$PATH

• Is the executing the correct one?o which echo  o type  echoo alias

Troubleshooting Ethernet

• Using networking or network-manager services• ifconfig, ip• route, traceroute, mtr• Configuration Files:

o /etc/network/interfaces (Ubuntu)o /etc/resolv.confo /etc/hosts

Troubleshooting Wireless

• Know your kernel: uname -r• Know your hardware: dmesg, lspci -v, lsusb -

v, /etc/modules• Know your configuration: ifconfig, iwconfig, nm-

tool, iwlist wlan0 scan• System, Preferences, Network Connections• http://www.hpl.hp.com/personal/

Jean_Tourrilhes/Linux/Wireless.html

NetworkManager

• If you use NetworkManager, then you don't use the networking service.

• nm-applet runs in notification area• No longer need to login to activate connections• nm-tool  provides useful information• Right click, Edit Connections• sudo restart network-manager

Troubleshooting Hardware

• Swap it! OS then parts• dmesg• lspci -v• lsusb• lshal• /proc

o cpuinfoo meminfo

Troubleshooting Disks

• fdisk -l• blkid• df -h, du -sh• mount• /etc/fstab• fuser -v foo.bar• lsof (-u user) (-p pid)• smartctl -a /dev/sda

Disk Recovery Tools

testdisk • recovers partitions and deleted files

foremost• recovers specific file types that are deleted

Troubleshooting Help

• Search for exact error message• Research• RTFM, info and doc• Use community forums• Use mailing lists• help.ubuntu.com• ubuntuforums.org• linuxquestions.org• freenode.net

References

• http://delicious.com/keithwright/scalets• http://help.ubuntu.com• http://ubuntuforums.org/• http://forums.fedoraforum.org/• http://tldp.org/• http://www.linuxtroubleshooting.com/• Irc freenode.net #rhel #fedora #ubuntu

Building Debian Packages

Debian souce code packages are distributed in .dsc files. To build any Debian package, the build environment must be installed through the metapackage build-essential.

            # apt-get install build-essentialFor each specific package foo, then:

# apt-get build-dep foo# apt-get source --build foo

Then create a dedicated version of your own builddch -l local 'Blah blah blah' debuild -us -uc Install your package by running sudo dpkg -i ../*.deb

Using the dpkg command

To enable logging of commands use the --log optionList all packages: dpkg -lTo query an individual package: dpkg --status packageViewing package status:  dpkg --get-selections \* > selections.txtSetting package status: dpkg --set-selections  <selections.txtInstalling a package: dpkg --install package.debUnpack a package: dpkg --unpack package.deb Configure a package: dpkg --configure packageRemove a package (but not configs): dpkg --remove packagePurge all package files: dpkg --purge packageList files in a package: dpkg --listfiles package Find package providing file: dpkg --search filename To find a package to provide a file: apt-file search filename    

apt-get - To install, update, remove, or purge packagesRepositories configured with /etc/apt/sources.listUpdate list of available packages: apt-get update Install updated packages without adding/removing: apt-get upgradeInstall all updated packages: apt-get dist-upgrade Install a specific package: apt-get install packageRemove a package: apt-get remove packagePurge a package: apt-get --purge remove package The cron-apt package creates a cron job to automatically update   

apt-cache - To query packages

To search for a keyword: apt-cache search keyword To query information: apt-cache show package To determine dependencies: apt-cache depends packageTo show detailed information about all versions of a package and its

dependencies: apt-cache showpkg package

Troubleshooting Packages

Try creating a new user and running the application. If it works for the new user, but not the old one, then old configuration files may be the problem. Move the old user's dot files and retry.

If one package overwrites (conflicts) with another packages files, then remove one of the packages.

If install or remove scripts for a package fail, then correct the problem in: /var/lib/dpkg/packagename.scriptname file.

Then use: dpkg --configure –a

apt-get update; apt-get upgrade -f

Package Management Frontends

aptitude     - works CLI and well as an text application    - tracks dependencies and removes them when not needed dselect - works as a menu driven text applicationsynaptic - GUI applicationsoftware-center - GUI application appearing on Applications menu