Module 13 - Troubleshooting

LINUX SHELL SCRIPTINGSEC 210

Module 13Troubleshooting, Performance,

and Security

OBJECTIVES

• Describe/outline good troubleshooting practices

• troubleshoot common hardware/software problems

• Monitor system performance via command-line

• Identify and fix common performance problems

• Describe the different facets of Linux security

• Increase the security of a Linux computer

• Measures & utilities to detect a security breach

2

3

The

maintenance

cycle

TROUBLESHOOTING METHODOLOGY

• Monitoring: observing log files and running

performance utilities system to identify problems

and their causes

• Proactive maintenance: minimizing chance of

future problems

• e.g., perform regular system backups

4

TROUBLESHOOTING METHODOLOGY (CONT)

• Reactive maintenance: correcting problems when

they arise

• Documenting solutions

• Developing better proactive maintenance methods

• Documentation: system information stored in a

log book for future references

• All maintenance actions should be documented

• Troubleshooting procedures: tasks performed

when solving system problems

5

COMMON TROUBLESHOOTI

NG PROCEDURES

6

TROUBLESHOOTING METHODOLOGY (CONTINUED)

• Two troubleshooting golden rules:

• Prioritize problems according to severity

• Spend reasonable amount of time on each problem

given its priority

• Ask for help if you can’t solve the problem

• Try to solve the root of the problem

• Avoid missing underlying cause

• Justify why a certain solution is successful

7

RESOLVING COMMON SYSTEM PROBLEMS

• Three categories of problems:

• Hardware-related

• Software-related

• User interface-related

8

HARDWARE-RELATED PROBLEMS

• Often involve improper hardware or software

configuration

• SCSI termination

• Video card and monitor configuration

• All hardware is on Hardware Compatibility List

• POST test alerts

• Loose hardware connections

• Problems specific to the type of hardware

• View output of dmesg command

• View content of /var/log/boot.log, /var/log/messages

9

HARDWARE-RELATED PROBLEMS (CONTINUED)

• Absence of device drivers prevent OS from using

associated devices

• dmesg command: displays the hardware that is detected

by the Linux kernel

• lsusb command: displays a list of USB devices detected

by the Linux kernel

• lspci command: displays a list of PCI devices detected

by the Linux kernel

• Compare outputs of commands to output of lsmod to

determine if driver module is missing from kernel

10

DMESG | GREP SDA

- list all detected devices

11

DMESG | GREP –I MEMORY

• search for lines having string memory

12

WATCH "DMESG | TAIL -20"

• real time dmesg monitoring

13

SUDO LSHW –CLASS MEMORY | LESS

• lshw classifies hardware using classes

14

LSPCI | LESS

• similar to dmesg | grep –i pci

15

LSMOD

• same as cat /proc/modules

• shows currently loaded kernel modules (LKMs)

• modules are pieces of code loaded into the kernel during boot (minus ext .o or

.ko… added with insmod)

• size of memory used by module

• used by: number of instances of module used

16

compare output

with dmesg, etc to

see if driver

module is missing


• Hardware failure can render a device unusable

• HDDs most common hardware components to fail

• If HDD containing partitions mounted on noncritical directories fails:

• Power down computer and replace failed HDD

• Boot Linux system

• Use fdisk (or LVM) to create partitions on new HDD

• Use mkfs to create filesystems

• Restore original data

• Ensure /etc/fstab has appropriate entries to mount filesystems

17


• If HDD containing / filesystem fails:

• Power down computer and replace failed HDD

• Reinstall Linux on new HDD

• Restore original configuration and data files

18

SOFTWARE-RELATED PROBLEMS:APPLICATION-RELATED

PROBLEMS

• Missing program libraries/files, process

restrictions, or conflicting applications

• Dependencies: prerequisite shared libraries or

packages required for program execution

• Programs usually check at installation

• Package files may be removed accidentally

19


PROBLEMS (CONTINUED)

• rpm –V command: identify missing files in a

package or package dependency

• dpkg –V bash (for ubuntu)

• ldd command: display shared libraries used by a

program

• ldconfig command: updates list of shared

library directories (/etc/ld.so.conf) and list of

shared libraries (/etc/ld.so.cache)

20

$LDD /BIN/BASH

• .so files are “shared object” files

• similar to Windows DLL

21



• Too many running processes

• Solve by killing parent process of zombie processes

• Filehandles: connections programs make to files

• ulimit command: modify process limit parameters

in current shell

• Can also modify max number of filehandles

• by default it’s 1024

• ulimit –n 5000 (increases file handles)

• ulimit –u 30000 (increases max # of user processes in

shell)

22



• /var/log directory: contains most system log files

• Some are hard linked to /var/log directory

• If applications stop functioning due to difficulty

gaining resources, restart using SIGHUP

• Do determine if another process trying to access the

same resources attempt to start application in Single User

Mode

• If resource conflict is the cause of the problem,

download newer version of app or application fix

23

SOFTWARE-RELATED PROBLEMS:OPERATING SYSTEM-RELATED

PROBLEMS

• Most software-related problems related to OS

• X windows, boot loader, and filesystem problems

• Problem detecting video card or monitors by the

kernel

• To isolate problem starting X Windows or gdm:

• View /var/log/Xorg.0.log file

• Execute xwininfo or xdpyinfo

24

SOFTWARE-RELATED PROBLEMS:OS-RELATED PROBLEMS

(CONTINUED)

• LILO (Linux Loader) problems: place “linear” in,

remove “compact” from /etc/lilo.conf file

• ubuntu: /etc/grub.d/10_linux, /etc/default/grub,

/boot/grub/grub.cfg

• GRUB problems: typically result of missing files in

/boot directory

• Ensure Linux kernel resides before 1024th cylinder

and lba32 (large block addressing) keyword is in

configuration file

• Eliminates BIOS problems with large HDDs

25


(CONTINUED)

• If filesystem on partition mounted to noncritical

directory becomes corrupted:

• Unmount filesystem

• Run fsck command with –f (full) option

• If fsck command cannot repair filesystem, use mkfs

command to re-create the filesystem

• Restore filesystem’s original data

26


(CONTINUED)

• If / filesystem is corrupted:

• Boot from Fedora installation media and enter System

Rescue

• At shell prompt within System Rescue:

• Use mkfs to recreate the filesystem

• Use backup utility to restore original data to the re-

created / filesystem (ie: tar, cpio, dump, restore, etc)

• Exit System Rescue and reboot system

• Knoppix Linux and BBC Linux: bootable Linux

distributions with many filesystem repair utilities

27

SOFTWARE-RELATED PROBLEMS: USER INTERFACE-RELATED

PROBLEMS

• Assistive technologies: tools that users can use to

modify their desktop experience

• Assistive Technologies Preference utility within GNOME

Desktop Environment

• Preferred Applications to configure Web browser,

multimedia player and terminal applications to be

opened automatically

• Mouse Accessibility to configure speed and click

behavior

• Keyboard Accessibility to configure keyboard

related assistive technologies

28

SOFTWARE-RELATED PROBLEMS: USER INTERFACE-RELATED PROBLEMS (CONTINUED)

29

Figure 14-3: The Assistive Technologies Preferences utility

PERFORMANCE MONITORING

• Jabbering: failing hardware components send

large amounts of information to CPU when not in

use

• Other causes of poor performance:

• Software monopolizes system resources

• Too many processes

• Too many read/write requests to HDD

• Rogue processes

30

PERFORMANCE MONITORING (CONTINUED)

• To solve software performance issues:

• Remove software from the system

• Move software to another Linux system

• Add CPU or otherwise alter hardware

• Bus mastering: peripheral components perform

tasks normally executed by CPU

31

PERFORMANCE MONITORING (CONTINUED)

• To increase performance:

• Add RAM (reduces swap time with HDD)

• Upgrade to faster HDDs

• Disk Striping RAID

• Keep CD/DVD drives on a separate HDD controller

• Run performance utilities on a regular basis

• Record results in a system log book

• Eases identification of performance problems

• Baseline: measure of normal system activity

32

MONITORING PERFORMANCE WITH SYSSTAT UTILITIES

• System Statistics (sysstat) package: contains

wide range of system monitoring utilities

• Use apt-get install sysstat command

• includes: mpstat, iostat, sar

33

MPSTAT

• multiple processor stats

• Used to monitor CPU performance

• Can specify interval and number of measurements rather than displaying

average values

• %sys should be smaller than %usr and %nice combined

34


(CONTINUED)

• iostat (Input/Output Statistics) command:

measures flow of information to and from disk

devices

• Displays CPU statistics similar to mpstat

• Displays statistics for each disk device on the system

• Output includes:

• Transfers per second

• Number of blocks read and written per second

• Total number of blocks read and written for the

device

35

IOSTAT | LESS36


(CONTINUED)

• sar (System Activity Reporter) command:

displays various system statistics taken in the last

day

• Provides more information than mpstat and iostat

• By default scheduled to run every 10 minutes

• Output logged to a file in /var/log/sa directory

• -f option: View statistics from a specific file

• Can be used to take current system measurements

37

sar –q 1 5

• # of processes in q. runq-sz (run queue size)

• 2 or less is normal for Intel architectures

38

sar 2 4

• 4 CPU stats taken every 2 seconds


(CONTINUED)

• Additional sar options:

• -q option: Displays processor queue statistics

• runq -sz value: Number of processes waiting for

execution on processor run queue

• plist -sz value: Indicates number of processes

currently running

• ldavg values: Represent average CPU load

• -W option: Displays number of pages sent to and taken from

swap partition

• Large number causes slower performance

• Add RAM to resolve

39


(CONTINUED)

40

Common options to the sar command

OTHER PERFORMANCE MONITORING UTILITIES

• top command: displays CPU statistics, swap usage,

memory usage and average CPU load

• free command: displays total amounts of physical

and swap memory and their utilizations

• Can be used to indicate whether more physical memory is

required

• vmstat command: displays memory, CPU, and swap

statistics

• Can be used to indicate whether more physical memory is

required

41

TOP – REAL TIME42

FREE & VMSTAT43

SECURITY

• Linux systems typically made available across

networks such as the Internet

• More prone to security loopholes and attacks

• Should improve local and network security

• Understand how to detect intruders who breach

the system

44

SECURING THE LOCAL COMPUTER

• Limit access to physical computer itself

• Prevent malicious users from accessing files by directly

booting the computer with their own device

• Server closet: secured room to store servers

• Remove floppy, CD, and DVD drives from

workstations

• Ensure BIOS prevents booting from USB ports

45

SECURING THE LOCAL COMPUTER (CONTINUED)

• Ensure BIOS password is set

• Set boot loader password in LILO (linux loader) or

GRUB configuration file

• Prevents intruder from interacting with boot loader

• Limit access to graphical desktops and shells

• Exit command-line shell before leaving computer

• nohup command: prevents background processes from

being killed when parent shell is killed or exited

• Lock screen using GNOME or KDE

46

SECURING THE LOCAL COMPUTER (CONTINUED)

• Minimize root user’s time logged in

• su (switch user) command: switch current

user account to another

• Used to switch between root user and regular user

• sudo command: perform commands as another

user if you have the rights to do that listed in

/etc/sudoers file

47

CAT /ETC/GROUP48

PROTECTING AGAINST NETWORK ATTACKS

• Always a possibility that hackers can manipulate a

network service by interacting with it in unusual

ways

• Buffer overrun: program information for a network

service altered in memory

49

NETWORK SECURITY ESSENTIALS

• Minimize number of running network services

• nmap (network mapper) command: scans

ports on network computers

• User can determine what network services are running

• Ensure that services that are not needed are not

automatically started when entering the runlevel

50

NETWORK SECURITY ESSENTIALS (CONTINUED)

• Ensure network service daemons for essential

services not run as root user when possible

• Ensure that shell listed in /etc/passwd for

daemons is set to /sbin/nologin

• Hacker will not be able to get BASH shell

• New network service versions usually include

fixes for known network attacks

• Keep network services up-to-date

51

NETWORK SECURITY ESSENTIALS (CONTINUED)

• TCP wrapper: program that can start a network

daemon

• Checks /etc/hosts.allow and /etc/hosts.deny files before

starting a network daemon

• Examine permissions for files and directories

associated with system and network services

52

CONFIGURING A FIREWALL

• netfilter/iptables: used to configure a firewall

• Discard network packets according to chains of rules

• Chains: specify general type of network traffic to apply rules to

• Rules: match network traffic to be allowed or dropped

• Three chain types:

• INPUT: incoming packets

• FORWARD: packets passing through computer

• OUTPUT chain: outgoing packets

53

CONFIGURING A FIREWALL (CONTINUED)

• iptables command: creates rules for a chain

• Can be based on source IP, destination IP, protocol used,

or packet status

• Stateful packet filter: Remembers traffic allowed

in an existing session and adjust rules

appropriately

• Easier to use graphical utility to configure

firewalls

54

55

Table 14-2: Common iptables options

CONFIGURING A FIREWALL (CONTINUED)

56

Figure 14-4: The Firewall Configuration utility

CONFIGURING SELINUX

• SELinux: Security Enhanced Linux

• By default, configured and enabled during Fedora installation

• Series of kernel patches and utilities created by NSA

• Enforces role-based security

• To enable, edit /etc/selinux/config file

• Configure SELINUXTYPE option

• Reboot and relabel the system

• sestatus command: view current SELinux status

57

USING ENCRYPTION TO PROTECT NETWORK DATA

• Use encryption algorithms to protect data before

it is transmitted on a network

• Asymmetric encryption: uses a pair of keys

uniquely generated on each system

• Public key: freely distributed

• Private key: used only by the system, never distributed

• Can be used to authenticate messages

• Digital signature: message that has been

encrypted using a private key

58

WORKING WITH SSH

• By default, SSH uses RSA to encrypt data and DSA to

digitally sign data

• System wide RSA and DSA key pairs are generated

the first time SSH daemon is started

• Tunneling: enclosing network traffic within encrypted SSH

packets

• SSH identity: used to automatically authenticate to

other computers using digital signatures

• Manage keys using Password and Encryption Keys

utility

59

WORKING WITH SSH (CONTINUED)

60

Figure 14-5: The Passwords and Encryption Keys utility

WORKING WITH GPG

• Open source version of PGP

• Each user has a key pair used for encryption and

authentication

• Authentication uses trust model

• Typically uses RSA and DSA key pairs for

asymmetric encryption and digital signing

• Can manage GPG keys and encrypt data using:

• gpg command

• Graphical utility such as Passwords and Encryption Keys

utility

61

DETECTING INTRUSION

• Log files can contain information or irregularities

indicating an intrusion

• Review log files in /var/log associated with network

services

• At minimum, review system log files associated with

authentication

• Pluggable Authentication Module (PAM): handles

authentication requests by network applications

• Log file in /var/log/secure

62

DETECTING INTRUSION (CONTINUED)

• Check /var/log/wtmp log file

• Lists users who receive BASH shells

• Use who command to view the file

• lsof (list open files) command: lists files

that are currently being edited

• Periodically search for files that have SUID bit set

• Tripwire: monitors important files and directories

• Intrusion Detection System (IDS): program used to

detect intruders on a Linux system

63

64

DETECTING INTRUSION (CONTINUED)

65

Table 14-3: Common Linux Intrusion Detection Systems

SUMMARY

• Administrators monitor the system, perform

proactive/reactive maintenance, and document

system information

• Common troubleshooting procedures involve:

• Isolating and determining the cause of system problems and

implementing and testing solutions that can be documented

for future use

• Invalid hardware settings, absence of device drivers,

and hard disk failure are common hardware-related

problems

66

SUMMARY (CONTINUED)

• Software-related problems can be application-

related or OS-related

• Users can use assistive technologies to modify

their desktop experience

• System performance is affected by a variety of

hardware and software factors

• Using performance monitoring utilities to create a

baseline is helpful for diagnosing future performance

problems

67

SUMMARY (CONTINUED)

• Securing a Linux computer involves:

• Improving local and network security and monitoring to

detect intruders

• Greatly improve local security by:

• Restricting access to the computer and using root account

only when required via su and sudo commands

68

SUMMARY (CONTINUED)

• Reduce chance of network attacks by:

• Reducing number of network services, implementing

firewalls, SELinux, service updates, encryption, and TCP

wrappers, and restricting services from running as root

user and permissions on key files

• Analyzing log files and key system files and

running IDS applications can be used to detect

intruders

69

Documents

Module 13 - Troubleshooting