SysAdmin Day Supplement

8/11/2019 SysAdmin Day Supplement

1/24

Start with machine data and Splunk software. End with an unfair advantage. 2014 Splunk Inc. All rights reserved.

www.admin-magazine.com

ADMINNetwork & Security

Digital

Special

Another greatcollection of simpletools for managing,

monitoring, andconfiguring your

Linux network

Bonus articles:ngrep: Easy and efficient network monitoring

hdparm: Tune up your hard disk or DVD drive

10MORETerrc

Admin Tools

US$ 7.95

fOR ThE Busy AdMin
http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.admin-magazine.com/http://www.splunk.com/download?ac=sysadminday2014


2/24

Start with machine data and Splunk software. End with an unfair advantage.

Splunk software lets you collect, analyze and transform machine-generated

big data into real-time insight. Proactively monitor and troubleshoot your

infrastructure end-to-end to avoid service degradation and prevent outages

so you can go home early.

Discover the worlds leading real-time platform for machine data.

Download Splunk for free today. www.splunk.com/download

2014 Splunk Inc. All rights reserved.
http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014http://www.splunk.com/download?ac=sysadminday2014


3/24

ADMINNetwork & Security

ADMIN Special

Editor in Chief Joe Casad

Managing Editor Rita L Sooby

Proofing and Polishing Amber Ankerholz

Layout / Graphic Design Dena Friesen, Lori White

Advertising www.admin-magazine.com/Advertise

Ann Jesse, [email protected]

Phone: +1-785-841-8834

Publisher Brian Osborn

Customer Service / Subscription

For USA and Canada:

Email: [email protected]

Phone: 1-866-247-2802

(toll-free from the US and Canada)

www.admin-magazine.com

While every care has been taken in the content of

the magazine, the publishers cannot be held

responsible for the accuracy of the information

contained within it or any consequences arising

from the use of it.

Copyright & Trademarks 2014 Linux New Media Ltd.

Cover Illustration Vladislav Kochelaevs, fotolia.comNo material may be reproduced in any form

whatsoever in whole or in part without the written

permission of the publishers. It is assumed that all

correspondence sent, for example, letters, email,

faxes, photographs, articles, drawings, are supplied

for publication or license to third parties on a non-

exclusive worldwide basis by Linux New Media

unless otherwise stated in writing.

All brand or product names are trademarks of their

respective owners. Contact us if we havent

credited your copyright; we will always correct any

oversight.

Printed in Germany

ADMIN ISSN 2045-0702

ADMIN is published by Linux New Media USA,

LLC, 616 Kentucky St, Lawrence, KS 66044, USA.

Table of Contents

pkstat 4Network monitoring works best

when the tool is functional but not

too complicated.

di 5A handy tool for displaying and

monitoring disk information.

Trickle 6View traffic stats and shoot down

programs that are taking too much

bandwidth.

GoAccess 7Study your logfiles in real time.

Mosh 8Remote access over slow networkconnections.

SSLScan 9The easy way to manage your SSL

servers.

PortSentry 10Identify and log port scans.

GeoIP Lookup 11Obtain geographical information on

domain names.

Whowatch 12Look for intruders with this process

watcher.

Snoopy 13Log terminal commands for futurereference.

Dear Readers:

Happy SysAdmin Day!The success of last years 10 Terrific Tools list got us

excited about another round. Read on for 10 more simple but useful tools

from the toolkit ofLinux Pro Magazinecolumnist Charly Khnast.

As a special bonus, were also including two more articles

describing other great tools for the busy admins toolkit:

ngrep 14Ngrep is a pattern-matching tool that separates the wheat from the chaff and doubles as a

lightweight packet sniffer.

hdparm 18Hdparm is the tool to use when it comes to tuning your hard disk or DVD drive, but it can also

measure read speed, deliver valuable information about the device, change important drive

settings, and even erase SSDs securely.

10MORETerrificAdmin Tools

10 More Terrific AdMin Tools!Wm

3M or e Gr e A T T ool s f or T he Bus y A d M i nW W W .A d M i n- M A GA zi ne .c oM
http://www.admin-magazine.com/http://www.admin-magazine.com/


4/24

Today, Im talk-

ing about a task

that isnt exactly

a big thrill for most administrators:

providing human-readable statistics

for traffic on a network interface.For this task, I recently discovered

pktstat [1] in the course of search-

ing for a compromise between the

monosyllabic IPTraf and the ver-

bose Wireshark. Pkstat is included

by most distributions, and the

source code is available online. To

see the current connections on an

interface, you simply type:

pktstat -i eth0

In a view that is remotely reminis-

cent of top, pktstat shows you the

network activity sorted by class

(ICMP, TCP, UDP, and so on). If

name resolution takes too long for

your liking, you can disable it by

setting the -nparameter. In the case

of protocols such as HTTP, FTP, and

X11, pktstat outputs more informa-

tion about the data transferred, such

as the path and the request methodfor HTTP (i.e., GETor POST).

Figure 1 shows the download sta-

tus for the ISO image of the future

Ubuntu LTS version 12.04. You

might notice that pktstat doesnt

show the full names of the source

and target machines only the bit

up to the first dot to ensure read-

ability. If you really want the whole

name, you need to enable the -F

parameter in pktstat.

You tend to lose visibility when

things start to liven up on a net-

work interface. To keep pace, you

can resort to two tweaks. For one

thing, after 10 seconds, pktstat de-

letes from its overview those con-

nections for which no data has

been transferred. You can reduce

this value to one second using the

-k(keeptime) parameter.Additionally, pktstat updates its

overview every five seconds. Speci-

fying -w 1speeds it up and refreshes

the view every second. The -wpa-

rameter can be used in another

way: pktstat offers a single-shot

mode, which you enable like this:

pktstat -i eth0 -1 -w 10

The -1parameter initiates single-

shot mode. Pktstat will run without

screen output for the number of

seconds specified in -w 10. It then

quits and leaves you a tidy over-

view of the connections it identified

as its legacy.

Re-Sorting

The tool offers some other parame-

ters for influencing the output; the

one I use most frequently is -l(lastseen). This tells pktstat to sort the

overview to

show me the

connections

that were last

active. The lon-

ger a communi-

cation is idle,

the farther

down the list it

slides. The -t

parameter (top

mode) will push

data streams

that shovel the

largest volume of data through the

interface to the top of the list. Most

command-line parameters also work

interactively at pktstat run time; you

can press the Lkey to enable last-

seen mode in this way.After working with pktstat for a

while, I think you will agree that it

provides administrators an uncom-

plicated approach to discovering

the traffic situation on their net-

works. For the classic question

Which process is currently grab-

bing all of the available band-

width? well, if you want to do

some detective work, you still

need Wireshark. n

Info

[1] Pktstat:http://www.adaptive-enterprises.

com.au/~d/software/pktstat/

The Author

Charly Khnast is a Unix operating system

administrator at the Data Center in Moers,

Germany. His tasks include firewall and DMZ

security and availability. He divides his lei-

sure time into hot, wet, and eastern sectors,

where he enjoys cooking, freshwater aquari-ums, and learning Japanese, respectively.

Wh t cm t ay tak uch a mtg twk tac, amtat hu

ch a t that ucty ucta a t t cmpcat. By Chay KhatTraffic Spotting

Figure 1:pktstat was bound to notice me downloading a whole ISO image.

The source and target host names are deliberately curtailed. H

annuViitanen,

123RF.c

om

pktat10 More TerrifiC AdMin Tools!

4 M o re G r e AT Tool s for T he Busy A d M i n W W W.A d M i n- M AGA zi ne .Co M


5/24

To be fair,I have to admit that

many two-letter commands com-

pensate for their compact size

with a breathtaking number of pa-

rameters. The tool I look at today,Di [1], is no exception. The name

stands for disk information its

a kind dfon steroids. Like its role

model, Di delivers information

about filesystems, but with much

more detail, and the output filters

are much better.

Figure 1 shows the output from

di -a, a list of all mounted filesys-

tems, including filesystems that do

not exist physically but that the

kernel hallucinates into the direc-

tory tree. The parameter -xlets

you specify filesystems you want

Di to hide (e.g., di -a -x proc

keeps the /procentry from being

listed). You can also specify multi-

ple filesystems in a comma-sepa-

rated list:

di -a -x proc,tmpfs,fuse

Di is clever enough to interpret

fuseas fuse*; thus, my fusectl

type filesystem mounted in /sys/

fs/fuse/cois hidden in Figure 2.

However, you can also turn this

around: The -I ext4parameter lets

you tell Di to list only ext4 filesys-

tems. Using a comma-separated

list, such as -I ext3,vfat,proc,

will work, too.

Machines as Readers

The example in Listing 1 shows

the basic information for my

(only) ext4 partition; however, of

all this information, I am only in-

terested in the

filesystem usage

stats as a percentage 19 percent

in this case. The -fswitch is a par-

ticularly useful option if you wantto process the output in a script. If

I just change the command line

slightly,

di -dH -I ext4 -n -f p

it returns a neat and compact 19%.

The -nparameter suppresses the

line with the headings; -f pre-

stricts the output to the percentage

value. If I had typed an uppercase

P, incidentally, it would have given

me the percentage of free inodes.

A comma-separated list is also

useful for easy ongoing processing

of values. Di knows this and

switches to CSV mode if you ap-

pend -c:

# di -dh -I ext4 -n -c

/dev/sda6,/,"141.9G","19.9G","114.8G",U

19%,ext4

Admittedly, these more complex

Di command lines look pretty

much as though my cat has

walked across the keyboard, but

you can say that of other two-let-

ter tools, too. n

Info

[1] Di: http://freecode.com/projects/

diskinfo

Th m quty a cmma i u, th w tt it hu hav,

th u tw-ky cmma ik , mv, a i c atu. W k at

i, a pviuy itt-kw ptativ thi cub. By Chay Khat

Di Is All In

Listing 1:i -H -I xt4

Filesystem Mount Size Used Avail%Used fs Type

/dev/sda6 / 141.9G 19.9G 114.8G 19% ext4

Figure 1:The di -acommand displays all filesystems, including the kernel pseudo-filesystems.

Figure 2:The -xparameter excludes specific filesystem types.

10 More TerrIfIC AdMIn Tools!i

5M or e Gr e A T T ool s f or T He Bus y A d M I nW W W .A d M I n- M A GA zI ne .C oM


6/24

I am over 40years old and am

starting to mellow in my old age.

No, Im only joking; certain phe-

nomena still drive me up the wall.For example, when I am using

SSH on a server to edit a configu-

ration file and the bandwidth is

so pathetic that the landing zone

is a matter of luck when you try

to position the cursor that really

makes me mad.

I know, I know, today even a

line to a Black Forest village has

enough bandwidth for an SSH

connection, if you have exclusive

access. Be-

cause hell, as

Sartre already

knew, is other

people: In my

case, its the

HTTP connec-

tions that are

pushing my

poor little SSH

to the edge. I

could turn toMosh [1], but

that helps with shaky connections

rather than crowded lines. My

remedy for traffic jams goes by the

name of trickle [2] [3].This traffic-shaping tool uses

LD_PRELOADto redirect some standard

library calls, such as socket()and

therefore only works with dynami-

cally linked binaries. However, that

practically includes all programs

that the typical user deploys to eat

up bandwidth. In the simplest case,

I might even be one of these users

myself; then, I can practice self-re-

straint when calling traffic-produc-

ing programs. To this end, I can start

Firefox, for example, with:

trickle -u 32 -d 256 firefox

This command limits the upload

speed (-u) to 100KBps and the

download speed (-d) to 300KBps.

Beware: These are actually kilo-

bytes, not bits. I can also reduce

speed in one direction only, if I am

not worried about the other direc-tion. Figure 1 shows the successful

application of a

download limit

to 2Mbps.

LateThrottle

Trickles boons

naturally only

occur to me

when the

download is al-

ready running

and the SSH

session hangs. Luckily, trickle has

a daemon mode. Therefore, I

launch trickle when I boot the ma-

chine with

trickled -u 32 -d 256

The values must be adapted to

match the available bandwidth.

When launched, the trickle dae-

mon searches for /etc/trickled.

conf, which can look like Listing 1.

It assigns certain protocols a prior-

ity and does some tweaking.

The values that follow

Time-Smoothingand Length-Smooth-

ingdetermine how great the fluctu-

ations can be over a certain inter-

val. The smaller the value, the

greater the benefits are for interac-

tive protocols such as SSH. With

larger values, sometimes a major

outlier is permissible in both up-

ward and downward directions.

Trickle has some disadvantages

compared with real traffic shaping,

but its fine for home use maybe Ireally am getting soft. n

Info

[1] Charlys Column: Mosh by Charly

Khnast. Linux Magazine, November

2013, pg. 46: http://www.linux-

promagazine.com/Issues/2013/156/

Charly-s-Column-Mosh

[2] Trickle: http://monkey.org/~marius/

pages/?page=trickle

[3] Traffic shaping with Trickle by Oliver

Frommel. Linux Magazine, January

2006, pg. 70:

http://www.linux-magazine.

com/Issues/2006/62/

Traffic-shaping-with-Trickle

I yu ata taic u m cngtin at tim, nt wy. Nw yu can ht

wn pgam that a havy n taic t up th inw an utw. By Chay KhnatBlown Away

Figure 1:Speedometer shows how a download succeeds at a

speed of around 2Mbits per second.

[ssh]Priority = 1Time-Smoothing = 0.1Length-Smoothing = 1

[ftp]Priority = 5Time-Smoothing=3Length-Smoothing=5

[www]Priority = 10Time-Smoothing = 8

Length-Smoothing = 10

Listing 1: tick.cn

Tick10 More TerrIfIC AdMIN Tools!

6 M o re G r e AT Tool s for T he Busy A d M I N w w w.A d M I N- M AGA zI Ne .Co M


7/24

A system adminschoice of weap-

ons for dueling with the daily grind

is likely to be pretty conservative.

For example, Webalizer has beenmy tool of choice for analyzing web

server logs for something approach-

ing eternity. However, there is no

shortage of alternatives: AWStats,

AWFFull, and others are available

for adventurous admins. But, why

experiment? These tools just do the

same thing that is, create intuitive

evaluations from web server logs.

I am tempted to be unfaithful,

however, if I need a real-time sum-

mary, with precision down to a sec-

ond. Apachetop gives you a line on

this, and I wrote about it some time

back [1]. Since then, a better tool

has hit the market: Go Access [2].

This tool parses the web server log-

file, evaluates it up to the present

point in time, and displays the re-

sults at the command line. Go Ac-

cess reads typical logfiles in Com-

mon Logfile format, but also in

Combined Logfile format. If youhave something more exotic, you

have the option of teaching Go Ac-

cess how to handle it.

Go Access writes its output contin-

uously, so I can watch the web server

work in real time. In the simplest

case, you just pass in one parameter,

the path to the logfile, at run time:

goaccess -f /var/log/apache2/access.log

Another practical aspect is the

ability to add an IP address and, at

the same time, tell Go Access to ig-

nore access to it:

goaccess -f /var/log/apache2/U

access.log-e 10.50.1.25

This approach avoids evaluating ac-cess by the monitoring systems

(Nagios, Icinga, or load balancers),

all of which cyclically check whether

the server is still alive.

Forward Roll

The command-line display is di-

vided into sections Go Access calls

modules. The modules are listed

from the top down, so you need to

scroll to see them all. Figure 1 only

shows the first four modules and the

first line of the fifth. However, Go

Access displays 11 modules, includ-

ing overviews of the most com-

monly used browsers and client op-

erating sys-

tems, the most

frequently re-

ferring sites,

and search

keys thatprompted

search engines

to point users

to my website.

Another

practical thing

is that Go Ac-

cess painstak-

ingly differenti-

ates between

crawlers and

genuine brows-

ers in its evalu-

ations. A mod-

ule only shows

the Top 6 list for its category. More

details are quickly accessed, how-

ever: Each module is represented by

a number in the header; for exam-ple, 6 - Operating Systems. If you

press 6on the keyboard and then O

(for open Detail View), you are

treated to a full list view. Pressing F1

displays more interactive shortcuts.

Authoritative reports on closed

logfiles are naturally part of Go Ac-

cesss feature set. It looks like Ill

be sending Webalizer and Apa-

chetop off to the happy hunting

grounds soon. n

Info

[1] The Sysadmins Daily Grind: Webalizer

Xtended by Charly Khnast, Linux Maga-

zine, February 2006, pg. 65

[2] Go Access: http://goaccess.prosoftcorp.com

Jut a a catma uky t pucha a w ag g vy

mth, y am a uky t chag th t a tut t

vy t. Cumt Chay Khat tch th cvatv phphy

th mth, u by th cham a w g t. By Chay Khat

Sweet Logger

Figure 1:Four of 11 real-time-capable Go Access modules.

10 More TerrifiC AdMin Tools!G Acc

7M or e Gr e A T T ool s f or T he Bus y A d M i nw w w .A d M i n- M A GA zi ne .C oM


8/24

Mh10 More Terrific AdMin Tools!

8 M o re G r e AT Tool s for T he Busy A d M i n w w w.A d M i n- M AGA zi ne .co M

I am writingthis column on the

Costa Brava and currently dang-

ling my feet in the Med. Thisstretch of coast is aptly named;

brava can be translated as

wild, inhospitable. Unfortuna-

tely, this description also applies

to Internet coverage beyond the

tourist beaches although WiFi

is ubiquitous in hotels, camp-

sites, and bars. At the moment,

Im using a network operated

by the Xiringuito beach bar

near the picturesque ruins of the

ancient Greek trading exclave of

Empries, and the connection is

pretty brava.

This situation is not going to

spoil the sunny afternoon for me,

however, because I still have

Mosh [1] stashed away as an ace

in my beach bag. The SSH re-

placement consists of a client

component and a server compo-

nent along with a wrapper script.

Initially, Mosh connects the clientand server via SSH on port 22 in

the normal way. Then, the server

hands the client a key, with

which it identifies itself hence-

forth, and Mosh drops the TCP

connection.

At this point, the client and ser-

ver talk only on UDP, using a port

in the range between 60000 and

61000 by default. I can use the

--port=parameter to

force Mosh to prefer a specific

port. UDP connections are very

robust; they even survive client

suspend phases.

Whats even better is that, be-

cause the client uses the key initi-

ally received from the server toidentify itself, it can even switch

IP addresses. So, if the beach cafe

network collapses and I swap to

smartphone tethering, my Mosh

session continues unfazed, and

my seaside reverie is undisturbed.

Token of Appreciation

If the only available connection is

unstable, this can lead to the

known issue that SSH does not

show you what you typed at the

terminal until the TCP connection

recovers. Although Mosh cant

work miracles in this case, it is cle-

ver enough to guess what the ter-

minal should be displaying, and it

sends the characters for output

just in case. Synchronization via

UDP continues to run in the back-

ground.

Thanks to Moshs predictivemechanism, working at the com-

mand line is a much smoother ex-

perience for me than using SSH.

Mosh also doesnt leave you in

the dark about what has actually

been transferred and what bytesare just predicted: The characters

that the Mobile Shell predicts are

underlined (Figure 1). So, if I want

to see the whole truth, I can disable

the prediction function with

--predict=never. Equally, I can

force prediction using

--predict=always. The default beha-

vior is a compromise: Mosh mea-

sures the latency of UDP connec-

tion in the background and swit-

ches on the predictive function if

the connection quality deteriorates.

Mosh has become indispensable

for me on the road. It cannot com-

pletely replace SSH, because it cur-

rently does not support X11 or port

forwarding and only speaks IPv4.

However, the developers are wor-

king on IPv6 as well as on an app

for Android mobile phones, which

is due for release on some other

sunny day.n

Info

[1] Mosh: http://mosh.mit.edu

dagg yu g th a wh jyg th Mtaa uh a

at th ppt a g itt t; tuaty, chay kw

what t . By chay Khat

Shell on the Beach

Figure 1:Mosh underlines characters that have not been transmitted because of a poor connection.


9/24

SSL-secured servicesare the

rule today, rather than the excep-

tion. But, how can I quickly and

easily check a large number ofservers to see whether the en-

cryption methods in use are still

up to date? With the SSLScan

tool [1].

In the simplest case, I can just

call SSLScan with the URL of the

website that I want to test: sslscan

example.com. Listing 1 shows thatSSLScan simply tried a long list of

ciphers and returned a status of

Accepted, Rejected, or Failedfor

each one.

However, I am

primarily inter-

ested in what ci-

phers the server

accepts, not what

it rejects. The fol-

lowing command:

sslscan --no-failed U

www.example.com

helps me signifi-

cantly thin out the

output, reducing it

to a third of the

original length.

Things become

even clearer if I

add more restric-tions. For example,

if I want to know

whether the server

still supports

SSLv2, I can check

the target like this:

sslscan --no-failed U

--ssl2

www.example.com

The --ssl3and

--tls1parameters

work in the same

way; however,

SSLScan also lets you test mail serv-

ers not just web servers. You need

the --starttlsparameter to do

this. Figure 1 shows the output from

sslscan --no-failed --starttls

--tlsv1kuehnast.com:25

The last column of the figure shows

which ciphers the server prefers.

Redirection

I can use --xml=to re-

direct the output to an XML file.

This method is useful for a script

with which I periodically check

and/or document the encryption

capabilities of the server. A combi-

nation with --targets=

is useful here. I can use this to

write a list of host names to the

file along with the port numbers,

if there happen to be any ports

other than 443. SSLScan then au-

tomatically checks the machines

one after another.Another addition to my toolbox!

The SSLScan security checker is

fast, lean, and easy to automate.n

Info

[1] SSLScan: http://sourceforge.net/

projects/sslscan/

I, lik Chaly, yu maag SSL-scu svs, a t iscv a tl that

yu will iitly appciat. It chcks whth th cmplt scuity stup is

up t at. By Chaly Khast

Keychain for Life

Figure 1:Charly uses SSLScan to check his mail server.

Listing 1:sslsca xampl.cm

01 Supported Server Cipher(s):

02

03 Failed SSLv3 256 bits ECDHE-ECDSA-AES256-SHA384

04 Accepted SSLv3 256 bits ECDHE-RSA-AES256-SHA

05 Rejected SSLv3 256 bits ECDHE-ECDSA-AES256-SHA

06

10 More TerrIfIC AdMIn TooLS!SSLSca

9M or e Gr e A T T ooL S f or T he BuS y A d M I nw w w .A d M I n- M A GA zI ne .C oM


10/24

Scanning the portson a machine

belonging to someone else is not

generally regarded as an attack.

Of course, any serious attack will

be preceded by a port scan. Ad-

mins who take security seriouslyalways take a proactive approach

to port scans, such as blocking the

IP address that initiated the scan

for an extended period of time.

PortSentry [1] lets you do this and

is included in most distributions.

The daemon identifies and logs

port scans and runs commands af-

ter doing so. The detection mode

is set in /etc/default/portsentry:

TCP_MODE="tcp"

UDP_MODE="udp"

If you dont want PortSentry to mon-

itor UDP ports, just delete the second

line. If you replace tcpand udpwith

stcpand sudp, the tool is more sensi-

tive to stealth scans. If you enter atcp

and audp,it binds all unused ports

below 1024 and reports them to the

attacker as open; this means that the

attacker knows just as much aboutyour system after the scan as before.

The /etc/portsentry/portsentry.

conffile gives you more scope for

setting up the system. Here, you can

define trigger ports that act as port

scan detectors. The default selection

is very useful; I would only change

it if I were running a daemon on one

of these ports.

It is more important to set the

sensitivity with the SCAN_TRIGGER

variable. The default of 0means

that PortSentry reacts immediately

if a trigger port is addressed. Val-

ues of 1or 2reduce the sensitiv-

ity and thus avoid false positives.

ADVANCED_EXCLUDE_TCP=does the

same thing: Ports that are often

addressed by external hosts, such

as Ident (port 113) and NetBIOS

(port 139), are excluded in atcpmode; similarly ADVANCED_EXCLUDE_

UDP-excludes the UDP ports 67,

137, 138, and 520 (DHCP, Net-

BIOS, RIP) (Figure 1).

By default, PortSentry doesnt

respond to scans but simply logs

their existence. You can modify

this behavior with:

BLOCK_UDP="0"

BLOCK_TCP="0"

A 1here prevents IP addresses that

have issued port scans in the past

from opening connections by telling

PortSentry to issue

/sbin/route add -host $TARGET$ reject

which drops the connections and re-

turns a refused message (Figure 1).

The IP address that issued the port

scan is logged in /var/lib/portsen-

try/portsentry.blockedand stays

there until you restart the daemon.

Securing Your Weapons

To prevent your own systems from

falling foul of PortSentrys traps, you

have the /etc/portsentry/portsen-

try.ignore.staticfile, which is

where you define individual hosts

or whole networks that will not be

counterattacked. Incidentally, if you

set BLOCK_TCPand UDPto 2, Port-

Sentry will run the command that

you define as KILL_RUN_CMD this

could be something like issuing a

text alert, but it could just as easily

run the large-bore Metasploit

weapon for vicious counterattacks.

A word of caution: Pointing a dou-

ble-barreled shotgun at somebody

who knocks at your front door is

generally regarded as unfriendly. n

Info

[1] PortSentry:http://sourceforge.net/

projects/sentrytools/

T cbat 10 yas his cum, Chay sts up a ssitiv tct that

masus th csmic backgu aiati th Itt. By Chay KhastTen Years After

Figure 1:PortSentry initializing and detecting port scans in line with its configuration.

Xxx PtSty10 More TerrIfIC AdMIn ToolS!

10 M o re G r e AT Tool S for T he BuSy A d M I n w w w.A d M I n- M AGA zI ne .Co M


11/24

All popular distributionsinclude

one or more packages that identify

the country of origin of an IP ad-dress. On my Ubuntu lab machine,

I use the geoip-binand geoip-data-

basepackages. Now, you can also

use the geoiplookupcommand and

geoiplookup6for IPv6 addresses,

with an IP address or a name as a

command-line parameter:

$geoiplookup linuxfoundation.org

GeoIP Country Edition: US, United States

For most purposes, I just need to

map the IP address to a country.

My spam filters use this technique

to determine the top five spammer

domiciles on a daily basis. Figure

1 shows that this is Germany, but

this is likely because I grabbed the

screenshot on a Sunday. Germany

is very rarely in the top five during

the week.

If you need more granular reso-

lution that is, you dont justwant the country, but the city, re-

gion, or organization you can

use GeoIP data by commercial pro-

viders. Typing geoiplookup linux-

foundation.orgwould then revealthe following:

GeoIP Country Edition:US, United States

GeoIP City Edition, Rev 1: US, OR, U

Medford,N/A, 42.326500, -122.875603, U

813, 541

GeoIP ASNum Edition: AS3701 Oregon U

JointGraduate Schools of Engineering

A libapache2_mod_geoip module is

available for web servers. This

helps me direct users to the area of

the site localized for them based

on their origin.

Sorting by Country

To sort by country, I added the fol-

lowing to my httpd.conf:

GeoIPEnable On

GeoIPDBFile /usr/share/geoip/geoip.dat

You might also need to modify the

path. I then added the lines from

Listing

1 to my

.htaccessfile.

The accuracy of the geodetic

data is almost always good

enough, at least at the country

level, but exceptions just go to

prove the rule.

Cellular radio providers route

their HTTP traffic through man-

datory proxies. Depending on

the network load, the proxy

might be in a neighboring coun-

try, giving rise to suspicions of

mass emigrations. n

Th gba viag i big ugh t wat t i ut wh yu

i a mi hav t up camp. Chay a quick

IP-ba itucti t ggaphy. By Chay Khat

Land Ahoy!

Figure 1:Germany is the world champion! At least on this strange Sunday and for Charlys antispam system with its integrated GeoIP lookup.

Listing 1: .htacc Aiti

01 #IP Address of .de

02 RewriteEngine on

03 RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^DE$

04 RewriteRule ^(.*)$ http://www.example.com/de

05

06 #Everyone else sees the English page:

07 RewriteEngine on

08 RewriteRule ^(.*)$ http://www.example.com/en/

10 More TerrIfIC AdMIn Tools!GIP lkup

11M or e Gr e A T T ool s f or T he Bus y A d M I nw w w .A d M I n- M A GA zI ne .C oM


12/24

Every server

with an IP ad-

dress on the

Internet receives

uninvited visits

at some point. Theusual scans and scripted

carpet bombing simply bounce off

my machines thanks to clever fire-

walling, port knocking [1], and tools

like Fail2ban [2]. To keep attackers

from working around my defenses,

I use two rootkit hunters: Rootkit

Hunter [3] and Chkrootkit [4]. The

latter, unfortunately, accuses my

DHCP server of packet sniffing:

eth0: PACKET SNIFFER(/usr/sbin/dhcpd[28382])

This result is a known false positive,

which I ignore. As an interim report,

I can say that my

varmint hunters

have not seen any

prey thus far.

Nevertheless, I

occasionally go on

patrol to see

whether a server isbehaving strangely.

I use whowatch[5]

for this purpose,

which launches in

the terminal with a

process list; the sec-

ond column shows

the owner. In the

third column, Who-

watch tells me

whether the user is

local or logged on

via SSH, Telnet, or

in some other way.

For remote users,

this information is followed by the IP

address, and for local users, just :0.

Hotkey Control

I have two ways of navigating thisinformation: I can use the arrow

keys to select a line, press Enter,

and see a tree view of the associ-

ated processes, as shown in Figure

1. Pressing O(owner) hides or dis-

plays the process owner; pressing D

(details) creates a window with de-

tailed information for the process.

My second option is to type T

(tree view) to show all running pro-

cesses. In this tree view, too, press-

ing Dwill display more information.

PressingL(list of signals) shows me

the control signals that I can send to

the process, such as HUP, INT, TERM,

and in an emergency KILL. I can dis-

play the overall system status, partic-

ularly in terms of memory manage-

ment, by pressing S(sysinfo), which

tells Whowatch to display the total

load on the screen, in a style verymuch reminiscent of top(Figure 2).

I have never found anything dan-

gerous on my server patrols to date,

but I do like that warm, safe, and

cozy feeling. n

Info

[1] Fwknop: http://www.cipherdyne.org/

fwknop/

[2] Fail2ban: http://www.fail2ban.org

[3] Rootkit Hunter: http://rkhunter.

sourceforge.net

[4] Chkrootkit: http://www.chkrootkit.

org(in Portuguese)

[5] Whowatch: http://whowatch.

sourceforge.net

F patcua a, Chay ccaay pat h v fam a hut

w attack. H ha put tgth a at tbx f th jb. ByChay KhatOn Patrol

Figure 1:In the tree view, Whowatch shows admins all the processes on the system.

Figure 2:Is this top? No, its Whowatch showing the total load after the S key has been pressed.

Xxx Whwatch10 More TerriFiC AdMin Tools!

12 M o re G r e AT Tool s For T He Busy A d M i n W W W.A d M i n- M AGA zi ne .Co M


13/24

At work,Im sometimes plagued

by annoying gaps in my memory:

What exactly was the name of

that neat tool that I used to flashthe LEDs on a specific network

adapter to help me find the NIC

in the rack? Or: How exactly did I

delete all files that were more than

a week old in a directory? The

answer to all of these questions is

in the Bash history, but Murphys

Law dictates that the history is

always a little bit too short. And,

in my case, theres another degree

of uncertainty: Which server did I

do this on?

Snoopy potentially offers a solu-

tion. The small library with the

dogs name, wraps around ex-

ecve()and always wakes up when

the computer runs a command.

Many distributions have Snoopy in

the pen, but if not, GitHub [1] will

help you out.

To enable Snoopy at boot time,

you need an entry in /etc/ld.so.

preload. I added the following line://snoopy.so. The path is typ-

ically lib. If you are building

Snoopy yourself, the library is

likely to be found in /usr/local/

lib/or something similar.

Building Snoopy yourself offers

some benefits. For example, you

can edit the snoopy.hheader file in

the source up front. If you enter

#define ROOT_ONLY 1

Snoopy only logs commands that

run with root privileges, but if you

install the tool from the distribu-

tion repositories, this option is not

set, and it logs any old command

no matter who ran it.

Unless configured to do other-

wise, Snoopy writes to /var/log/

auth.log. Figure 1 shows the log for

some simple commands. The struc-

ture always stays the same; each

entry starts with the user ID, fol-

lowed by the session ID and the

TTY you use. This is then followed

by the working directory, which is

important because Snoopy does not

log commands like cd /etc. Navi-

gating the system is not the same

for this dog as executing a file.

This information is followed bythe full path to the executed file

and, finally, the expanded com-

mand (e.g., aliases can cause an

expansion). Many distributions

run ls --color=auto, so, in this

case, if you only type ls, Snoopy

reveals all.

Collection PointNow you just need to consolidate

the logs centrally. I configured one

server to accept the log messages

from other machines. If the server

runs rsyslog, you can just pass in

the -rparameter at boot time to

switch rsyslog to receive mode.

Next, you can tell your other serv-

ers also to send entries in /var/

log/auth.logto the newly config-

ured syslog server. To do this, you

just need to add one line to the

syslog configuration:

auth,authpriv.* @

The auth log tends not to grow

drastically, which means you can

rotate on a weekly or even monthly

basis. Snoopy fills a substantial log

of my heroic deeds of administra-

tion day after day including typosand similar peanuts. n

Info

[1] Snoopy: https://github.com/a2o/

snoopy

Smtms sys am Chay s t kw wh xacty h smthg

gus f hs svs. Fg a fab mmy a s ffcut,

yu mght thk. Pauts! says Chay. By Chay Khast

Guide Dog

Figure 1:A neatly maintained history thanks to Snoopy.

10 More TerriFiC AdMin ToolS!Spy

13M or e Gr e A T T ool S F or T he BuS y A d M i nw w w .A d M i n- M A GA zi ne .C oM


14/24

You might want to inspectyour

network at a very detailed level

for a number of legitimate rea-sons. Much of the time, its to

debug an application thats misbe-

having and connecting to a server

on the wrong port, or maybe a

colleague has noticed a slowdown

on a particular network link, and

you need to diagnose where the

sudden flood of multidirectional

traffic is coming from.

On the other hand, you might

need to check the exact nature of

an attack and perform some real-

time forensic diagnostics to cir-

cumvent it. Leaving the network-

ing aspects aside for a moment,

even an admin solely responsible

for systems and not networks

(an exceptionally rare remit these

days, admittedly) needs a highly

functional packet sniffer avail-

able at all times. Because systems

rely so heavily on connectivity

for multifaceted Internetusage, its

imperative for admins to be able

to inspect the contents of the

network deeply and interpret theresults proficiently.

The all-pervasive networking

tool tcpdump [1] is undoubtedly

still the champion of packet sniff-

ers but, for certain scenarios,

I much prefer an equally light-

weight package called ngrep [2],

sometimes called simply network

grep. As its name suggests, ngrep

does for networks what grep does

for files; its a highly functional

network pattern-matching tool

that helps the user sort the wheat

from the chaff, and on a busy net-

work, you will need a great deal

of assistance to determine what

the seemingly endless flood of

characters quickly running up the

screen actually means.

Whats the Difference?

When I first started looking atnetworks in any great detail, I

was initially attracted to ngrep

because its command structure

seemed to be in plain English. It

uses words, unlike this tcpdumpexample, which doesnt exactly

make sense at first glance:

# tcpdump -vv -i eth1 'tcp[13] & 2 = 2'

Matching either SYN only or SYN-ACK U

datagrams

The preceding example looks

more like a demonstration of

why I should have listened to

my mathematics teacher prop-

erly when I was still in school.

If youre familiar with regular

expressions, then youll know one

of the aspects that made tcpdump

so popular was its flexibility. On

the other hand, ngrep follows

the same path but appears to of-

fer more of a grep-style filtering,

which, having used grep fre-

quently, to my mind at least feels

more intuitive to use. However,

you dont need to be strictly pur-ist, and using both tcpdump and Le

adim

age

Jean-

LucGirolet,

123RF.c

om

Ngrep i pern-mching h r he whe frm he

chff n ube ighweigh pcke niffer. B Chri Binnie

Network grep

Thresher

ngrep: Newrk Grep10 HaNdy adMIN tools

14 M o re G r e at tool s for t He Busy a d M I N w w w.a d M I N- M aGa zI Ne .Co M


15/24

ngrep can provide a great deal of

invaluable functionality.

Words and Numbers

To begin, Ill look at some simplis-

tic filtering rules that make ngrep

so attractive. To access a network

interface fully, you will need el-

evated privileges (e.g., su- or sudo

-s) to fully achieve that status be-

fore running the examples below.

For those of you less concerned

with repetitive strain injury, simplyprefix sudoto your command lines.

If youre concerned about email

traffic and need to watch all TCP

traffic closely using the SMTP

port, then you could construct a

command line such as:

# ngrep -d any port 25

Here, the SMTP example shows

that (in more recent libpcap library

versions, at least) you can ask

ngrep to listen on all the available

interfaces at once; otherwise, you

might just specify -d DEVor, for ex-

ample, -d eth1instead to specify a

particular network interface.

Now, Ill expand on that first

command a little and add more

switches to the example. By omit-

ting the -d anyparameter, the

trusty ngrep will assume a default

interface, usually eth0. Just ap-pend it as above if the examples

that follow arent what you need.

You can drill down into any

HTTP traffic on your network

link by mentioning port 80. Ad-

ditionally, you can isolate one

sender IP address that is sending

the port 80 traffic. Notice the src

hostsyntax:

# ngrep port 80 and src host 12.34.56.78

Moving on from a single IP ad-

dress, imagine that you have so

much data from that single IP

address that you want to refine it

even further and specify a desti-

nation address, too. In this case,

your example would look like this:

# ngrep port 80 and U

src host 12.34.56.78 and U

dst host 98.76.54.32

The dst hostappendage followed

by the destination IP address is, I

hope, self-explanatory. If you see

fit, you can then easily interchange

the hostelement with net; if youuse the CIDR format [3], your com-

mand line might then look like

# ngrep port 80 and U

src net 12.34.56.0/24 and U

dst net 98.76.54.32/27

instead.

Master Class

By now, I hope you can see how

its possible to wade through even

the heaviest floods of network

traffic and still discern whats go-

ing on and from where. One of the

more granular functions of ngrep

is its ability to pick out certain

pieces of information quickly from

the deluge of data thats streaming

up the screen. For unencrypted

logins, this works a treat. I sin-

cerely hope its only in a LANenvironment that you are still us-

ing Telnet, but if you need to hunt

down the login prompt to a Telnet

server, you can use this:

# ngrep -t -wi "login" port 23

Running this command spawns

ngrep under the default network in-

terface and offers the following in-

formation in addition underneath:

filter: (ip or ip6) and ( port 23 )

match: ((^login\W)|(\Wlogin$)|U

(\Wlogin\W))

Here, ngrep is saying it will listen

for both IPv4 and IPv6 traffic on

port 23 for Telnet. The matchis the

pattern for which ngrep is search-

ing. The -wswitch tells ngrep

to match the regular expression

(login, in this case), and the -i

switch means ignore case sensi-

tivity on that regular expression.

If youre stopping a steady

flow of traffic shooting up your

screen with Ctrl+C, then its use-

ful to have a time reference when

youre scrolling back through thedata, and thats exactly what the

-tparameter should do, with

timestamps for each match in the

form: YYYY/MM/DD HH:MM:SS.

UUUUUU.

Flick a Switch

Before I look at more examples,

Ill take a breather and look a little

at the some of the other available

switches that ngrep supports.

If youre keeping a keen eye

on all network traffic, you might

even have the need to look at

empty packets, which are usually

discarded because they have no

actual payload through which to

search. By adding -eto the com-

mand, then despite the added

regular expression, you can still

catch empty packets on the net-

work, which could be of a mali-cious nature.

Conveniently, in the same vein

as the stalwart grep, you could

simply add -vto reverse the filter

to see packets that dont match

the prescribed pattern.

I mentioned using tcpdump in

hand with ngrep, and the -l

option works nicely for this. If

youve captured and saved a large

dump of network data to a file

with tcpdump, then you can run

ngrep over the top of that data file

and use its simple, yet powerful,

searching functionality to do so.

10 HaNdy adMIN toolsngrep: Newrk Grep

15M or e Gr e a t t ool s f or t He Bus y a d M I Nw w w .a d M I N- M a Ga zI Ne .C oM


16/24

Using the example from above,

you can search for Telnet logins

from within a pre-saved tcpdump

dump file:

# ngrep -wi "login" port 23 U

-I

By enabling -X, you can inform

ngrep that youre looking for a

hexadecimal pattern, and not

plaintext, which is useful for more

advanced searching.

Finally, how about dumping di-rectly from ngrep onto a text file of

your choice? Its a simple maneu-

ver and involves the -Oparameter.

The nice thing about this fea-

ture is that it allows you to see

all of the required data on your

screen and still store it in a pcap-

compatible data file for later (the

highly portable libpcap library

format).

Lead by Example

Next, you can gather this newly

found knowledge and apply some

of these switches to what will

hopefully prove to be useful exam-

ples. Some of these are available in

more detail on the ngrep website

if you get stuck or are curious, but

Ill cover a few others, too.

Returning to the Telnet login

example above, think about an un-encrypted and clear-text FTP login

sequence and how you might go

about pattern matching such a ses-

sion taking place on your network:

# ngrep -wi -t U

-d eth0 'user|pass' port 21

The FTP login session capture

is frighteningly simple, as is the

Telnet login capture, and high-

lights precisely why everything

for which you can justify a little

extra complexity is encrypted on

networks these days.

Apparently ngrep can also delve

into the payloads of packets us-

ing regular expressions. This next

regular expression looks at a pre-

recorded network dump file. One

such (untested by me) expression

to examine US social security

numbers could be:

# ngrep -t -O U

'~.*(\*|\[[^]]*)'

To spot an HTTP attack that in-

volves endless HTTP POSTcom-mands, you add a caret in front of

the regular expression,

# ngrep -t '^(POST) ' 'dst port 80'

which instructs ngrep to look only

for POSTs at the beginning of the

payload associated with the packet.

Well Refined

One of the most important fea-

tures of ngrep is its ability to sort

the wheat from the chaff. If you

looked at raw port 80 traffic, you

would see lots of useful informa-

tion, as well as lots of potentially

useless information that doesnt

help you decipher whats travel-

ling across your network link. The

following HTTP sniffing example

is going to be noisy in terms of

output,

# ngrep port 80

whereas the next example, which

uses the clever bylinefunctional-

ity, helps boil down the screeds of

information efficiently:

# ngrep -W byline port 80

The byline function is the epitome

of simplicity and wraps text when

a new line is spotted, making

those raw HTTP packets sig-

nificantly easier to read with the

human eye. It differentiates the

packet headers and their associ-

ated payload nicely, too.

Reaction Time

On my travels, I once came across

a useful tool called tcpkill [4].

In the past, I have used it on a

Linux router to drop specific con-

nections between hosts that are

unnecessary or malicious. It might

surprise you to know that ngrep

offers exactly that functionalitytoo; that is, it lets you capture and

disconnect certain network traffic,

disrupting the TCP flow between

hosts by sending a set number of

RSTs.

In this case, the ngrep manual

offers the following entry for the -K

parameter and mentions the tcpkill

tool as well: -K num Killmatching

TCP connections (like tcpkill). The

numeric argument controls how

many RST segments are sent.

The Beginning of the End

This bit of insight might tempt

you to turn to ngrep the next time

youre looking for something on

your networks. The clarity of its

output and its minuscule instal-

lation footprint make it an indis-

pensable tool.

I havent gone into any detail re-lating to ngreps formatting of bi-

nary (hexadecimal) traffic, but its

certainly impressive and, again,

uses a familiar grep structure.

Combined with its other function-

ality, ngrep is undoubtedly a force

to be reckoned with. n

Info

[1] tcpdump: http://www.tcpdump.org/

[2] ngrep: http://ngrep.sourceforge.net/

[3] CIDR format: http://en.wikipedia.org/

wiki/Classless_Inter-Domain_Routing

[4] tcpkill:

http://en.wikipedia.org/wiki/Tcpkill

ngrep: Newrk Grep10 HaNdy adMIN tools

16 M o re G r e at tool s for t He Busy a d M I N w w w.a d M I N- M aGa zI Ne .Co M


17/24

Shop the Shop shop.linuxnewmedia.com

FIND IT ON NEWSSTANDS NOW OR ORDER ONLINE:

shop.l inuxnewmedia.com/rpi

In case you missed

it last time...

You ordered your Raspberry Pi...

You got it to boot...what now?

The Raspberry Pi Handbook takes

you through an inspiring collection of

projects. Put your Pi to work as a:

media center

photo server

game server

hardware controller

and much more!

Discover Raspberry Pis special tools

for teaching kids about programming

and electronics, and explore advanced

techniques for controlling Arduino

systems and coding GPIO interrupts.

WATCH YOUR NEWSSTANDS FOR

THE ONLY RASPBERRY PI REFERENCE

YOULL EVER NEED!
http://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpihttp://www.medialinx-shop.com/rpi


18/24

In 2005, Canadian Mark Lord

developed the small hdparm util-

ity [1] to test Linux drivers for IDE

hard drives. Since then, the pro-

gram has developed into a valu-

able tool for diagnosis and tuning

of hard drives.

For example, it tests the speed of

hard drives and solid state disks,

puts devices to sleep, and turns

the energy-saving mode on or off.

With modern devices, it can acti-

vate the acoustic mode and clean

up SSDs.

Before your first experiments

with hdparm, you should read the

safety concerns in the Warning!

box.

Need for Communication

All reasonably new distributions

already include hdparm in the

basic installation. You only need to

open a terminal and call up

hdparm -I /dev/sda | more

as administrator (Figure 1).

The tool will deliver all available

data about the chosen drive in

this case, the first hard disk sda.

The | moreoption makes sure the

large amount of information does

not simply rush unread through

the terminal.

Hdparm accepts any device as

mass storage that is connected to

an (E)IDE, SATA, or SAS interface,

including, therefore, DVD drives

and SSDs. USB-to-IDE adapters

often cause problems because they

do not transmit the (complete)

ATA or ATAPI commands to the

drive.

The information that hdparm

delivers is dependent on the de-vice. The designation and firm-

ware version number are always

listed at the top underModel

NumberandFirmware Revision.

Owners of an SSD especially can

find out quickly whether they are

running the current firmware

version.

On newer hard disks, you

should check whetherNative Com-

mand Queuing (NCQ)is to be

found under Commands/features.

This technology makes it possible

for the hard disk to sort queries

from the system in such a way that

Hdpam s th t t us wh t ms t tug yu had dsk DVD dv, but

t a as masu ad spd, dv vauab mat abut th dv,

hag mptat dv sttgs, ad v as SSDs suy. By Tm Shma

LeadImageA

myWalters,

123RF.com

Retrieving and setting hard drive parameters with hdparm

Disk Inspector

Wag

Hdparm manipulates a drive directly,

which is why using it can easily lead to

loss of data and, in the worst case, to a

defect on the device. Beyond that, the

programs documentation points out

that many of its functions are experi-

mental or dangerous. Therefore, before

you work with the program, you should

always make a backup of the complete

drive. Furthermore, you should only use

functions whose actions you fully under-stand. The publisher and author of this

article accept no liability for damages

or loss of data.

hdpam Dv Utty10 More Terrific ADMin ToolS!

18 M o re G r e AT Tool S for T He BUSy A D M i n W W W.A D M i n- M AGA zi ne .co M


19/24

the heads take the shortest possi-

ble path. SSDs, on the other hand,

distribute write accesses more effi-

ciently across storage blocks. Ide-

ally, this leads to an increase in

speed.

If NCQ is deactivated, check the

BIOS to find out whether the drive

is running inAHCImode, which is

also necessary for other functions

such as energy management.

Speedometer

To determine how fast a drive de-

livers data, you can use the

hdparm -t /dev/sda

command. After a few seconds,

the data transfer rate appears (in

megabytes per second, MBps).

The small program reads directly

from the drive for a while regard-

less of the filesystem. The speed

measured is therefore somewhat

faster than in actual practice. To

receive an untainted result, no

other programs should be running

during the measurement, and

enough main memory should be

free.

Repeat the measurement at least

three times and then calculate the

average value. For a current

model, the result should reach at

least 80 MBps (Figure 2).

The Linux kernel deposits the

data retrieved from the hard drive

into a buffer. To determine the

speed of the unadorned drive, you

can use the

hdparm -t --direct /dev/sda

command. Hdparm then reads the

data directly from the disk. The

values thus measured will be

somewhat slower than without

--direct, but at least you can see

the pure transmission rate of the

disk (Figure 3).

Hdparm always reads the data

from the beginning of the storage

device. Hard disks, however, tend

to deliver data somewhat more

slowly from the outer areas of

magnetic disks; therefore, hdparm

lets you set an offset (from soft-

ware version 9.29 on):

Figure 3:Without the buffer, transmission rate drops dramatically. At the middle of the 320GB hard

drive, more speed losses are seen.

Figure 1:Hdparm lists the hardware properties of a six-year-old hard disk with a 320GB capacity.

Figure 2:This SATA hard drive achieved an average read speed of 80.48 MBps.

10 More Terrific ADMin ToolS!hdpam Dv Utty

19M or e Gr e A T T ool S f or T He BUS y A D M i nW W W .A D M i n- M A GA zi ne .c oM


20/24

hdparm -t --direct --offset 500 /dev/sda

The 500stands for the number of

gigabytes to skip. On a 1TB hard

disk, the command above would

therefore deliver data from the

middle of the disk. As Figure 3

shows, reading speed drops quite

markedly in the outer areas of a

hard disk.

All the speed tests introduced

here only give a first impression of

possible problems and bottle-

necks. For a complete benchmark,therefore, you would also need to

determine the write speed, for ex-

ample.

Faster, Faster

Some drive properties can be

changed while the device is in op-

eration; for example, most drives

allow you to turn power manage-

ment on and off. Which functions

hdparm can change and activate

on a hard drive can be called with

hdparm -I /dev/sda

and are found under Commands/

features(Figure 1). All functions

found there and marked with an

asterisk are currently active, and

hdparm can use the rest or at least

activate them.

To speed up data transmission, ahard disk usually reads several

sectors at the same time. How

many it can deliver at the same

time is revealed by

hdparm -I /dev/sda

and is listed afterR/W multiple

sector transfer: Max =. This value

should also be found in the same

line after Current =. If that is not

the case, you can increase the

value with:

hdparm -m16 /dev/sda

This instructs the hard drive al-

ways to deliver 16 sectors at once.

Curiously, some hard drives run

slower with higher values: The hd-

parm man page mentions primarily

older Caviar drives from Western

Digital. In such cases, you should re-

duce the number of sectors again or

even turn off the function with:

hdparm -m0 /dev/sda

Beyond this, modern drives can even

retrieve a few sectors in advance(read ahead). To define how

many, use the -aswitch (Figure 4,

top), for example:

hdparm -a256 /dev/sda

Here, the drive will read in advance

the 256 sectors that are most likely

to be requested next. Higher values

speed up the reading of large files

at the cost, however, that reading

smaller ones takes longer. The cur-

rent setting is shown with

hdparm -a /dev/sda

Beyond that, many drives also pos-

sess a built-in, additional read-ahead

function. As a rule, therefore, you

can leave the setting at the default

value. How fast queries from the op-

erating system reach the hard drive

controller can be called with

hdparm -c /dev/sda

The value should be 32-bit; you can

force this value with the -c3switch.

Full SpeedAhead

Many modern hard

drives allow you to

slow down the head

movement. Although

doing so will in-

crease access times,

it will also reduce the noise level.

To see if your own hard drive of-

fers this acoustic mode, use:

hdparm -M /dev/sda

If a number follows the equal sign,

as shown in Figure 4 (bottom), the

drive can be put into a quiet mode

with:

hdparm -M 128 /dev/sda

To reach the highest speed, use themaximum value:

hdparm -M 254 /dev/sda

Values between 128and 254are al-

lowed, resulting in a trade-off be-

tween noise level and speed. Inci-

dentally, your Linux kernel must

also support acoustic manage-

ment, which should be the case

for all current major distributions.

Some CD and DVD drives turn

out to be more like turbines: Their

high-speed rotation can hinder

audio/video enjoyment. The

hdparm -E 4 /dev/sr0

command will provide relief. The

parameter 4determines speed, and

/dev/sr0specifies the DVD drive.

This example slows drive reading

speed ninefold.

Write-Back Caching

With write-back caching, the hard

drive first stores the data to be

Figure 4:Here, the read-ahead is set to 256, and acoustic

management is currently deactivated.




21/24

written in a buffer. In this way, it

can accept data much faster,

which in the end leads to a faster

write speed. The

hdparm -W /dev/sda

command shows whether write-

back caching is active with a 1

after the equals sign; otherwise,

you can activate the function with

the -W1switch.

If hdparm will not allow this

change, you need to make surethat write-back caching has been

activated in the BIOS. However,

this function is not recommended

for all situations: In the case of a

power outage, the data in the buf-

fer would be lost permanently.

If a program sensitive to data

loss such as a database is run-

ning on the system, you should

turn off the write-back cache with

the -W0switch. Documentation for

the PostgreSQL database even ex-

plicitly recommends that this be

done.

Live Wire

If a hard disk or SSD doesnt have

anything to do for a certain period

of time, it automatically enters

sleep mode. This power-saving

feature can be influenced with the

-Bparameter. Thus, using:hdparm -B255 /dev/sda

would deactivate energy manage-

ment; however, not all drives

allow this.

Instead of 255, values between 1

and 254 are allowed. A higher

value means more power is used

but also promises higher perfor-

mance or speed. Values between 1

and 128 allow the drive to shut

down, whereas values from 129 to

254 forbid that from happening.

The most power can be saved

with a value of 1; the highest rate

of data transmission (I/O perfor-

mance) is achieved with 254. You

can call up the current value with:

hdparm -B /dev/sda

The specific effect the different

values will have depends on the

drive itself. However, you should

keep in mind that too many shut-

downs are not good for desktop

hard drives: Each time it shuts off,

the drive must park the heads,

which increases wear and tear.Consequently, you shouldnt wake

up your hard drive every two sec-

onds which always takes more

than two seconds to do.

You can set how many seconds

of idleness the hard drive should

wait before it goes to sleep with

the

hdparm -S 128 /dev/sda

switch; however, this value here is

not in seconds but a number be-

tween 1 and 253.

The hard drive multiplies this

value by another. The value cho-

sen in the example, 128, lies be-

tween 1 and 240, for which the

drive uses a factor of five. Conse-

quently, it would shut down after

640 seconds of idleness.

From 241 and up, the multiplica-

tion factor increases steadily. At251, the waiting period has in-

creased to 5.5 hours. At 253, the

value is preset by the manufac-

turer, usually between eight and

12 hours. The value 254 is left out;

at 255, the drive will wait 21 min-

utes and 15 seconds. A value of 0

will deactivate sleep mode com-

pletely. To send the hard drive to

sleep immediately, enter:

hdparm -y /dev/sda

With a capital Y, the drive will go

into an even deeper state of sleep.

Depending on the drive, the drive

might only wake up from a deep

sleep after a reset of the whole

system.

Cleanup

SSDs track the location of the data

deposited on them independently

of the operating system. This can

lead to the curious situation that a

file has been deleted but the SSD

still has its former location marked

as occupied.To remedy such conflicts, newer

versions of hdparm include the

wiper.shscript. Entering

wiper.sh /dev/sda

determines which blocks are being

used and which are not and re-

ports this to the SSD. However,

this script must be used with cau-

tion: The documentation warns

explicitly that data could be lost

and advises against its use with

the Btrfs filesystem.

Drives with ext2/3/4, Reiser3,

and XFS should be mounted as

read-only before using the wiper

command. It would be best to un-

mount the drive completely or

start wiper.shfrom a Live system.

In any case, you should definitely

make a backup of the SSD before-

hand and use the script only in anemergency. Incidentally, because

wiper is so dangerous, some distri-

butions do not even include it.

Secure Deletion

To achieve higher transfer rates

and spread use equally over the

storage chips, SSDs also reserve

some storage areas (wear level-

ing), so that simply formatting an

SSD will seldom delete the whole

drive. Most SSDs therefore offer a

function called secure erase, which

causes the drive to empty all its

10 More Terrific ADMin ToolS!hdpam Dv Utty

21M or e Gr e A T T ool S f or T He BUS y A D M i nW W W .A D M i n- M A GA zi ne .c oM


22/24

storage cells. This is especially

useful should you decide to give

up your used SSD.

Secure erase has two pitfalls: hd-

parm can only initiate a secure

erase when the BIOS also allows it.

Beyond that, the method is consid-

ered to be experimental. The docu-

mentation warns explicitly about

using the procedure because, in the

worst case, secure erase could

make the whole SSD unusable. If

you want to use this delete function

anyway, first call up the identifica-tion information with:

hdparm -I /dev/sdb

Under Security, the line supported:

enhanced eraseshould show up

somewhere; otherwise, the SSD

wont support secure erase. Next,

turn on the security function of the

drive by (temporarily) setting a

password like 123456:

hdparm --user-master u U

--security-set-pass 123456 U

/dev/sdb

When you call up the identifica-

tion information again, you will

now find enabledunder Security.

To erase the SSD now, enter:

hdparm --user-master u U

--security-erase 123456 /dev/sdb

In the process, hdparm also re-

moves the password. The whole

process takes a few minutes, de-

pending on the size of the SSD,

during which no feedback is given.

Afterward, when you call up the

identification information, the area

under Securityshould look like it

did before setting the password.

Relics

In the case of older hard drives

with an IDE connector (also called

PATA), you should take a look at

the using_dmaline in the identifi-

cation output.

With the help of DMA (Direct

Memory Access) technology, the

hard drive itself deposits data di-

rectly into main memory. If the re-

spective flag is 0 (off), it will slow

down the data transfer. Over the

years, ever faster DMA standards

have been introduced; the fastest

possible can be activated with the

command:

hdparm -d1 /dev/hda

On some very old systems, how-

ever, the DMA mode can cause

problems. After activating it, you

should therefore copy a few larger

test files to the drive.

If problems arise or the drive

crashes, you can deactivate the

DMA mode again with:

hdparm -d0 /dev/hda

Incidentally, modern SATA drives

always use DMA.

While the hard drive is transfer-

ring the requested data, the rest of

the system can go about complet-

ing other tasks but only if an on

appears after unmaskirqin the

identification info output. You can

force this mode with the -u1

switch.

Lasting Values

After restarting the system, all

changes made with hdparm are

lost. To activate them perma-

nently, the respective hdparm

commands must be entered in the

start scripts.

How this is done depends on the

distribution you are running, but

usually the entry must be made in

/etc/rc.local.

Debian-based systems, on the

other hand, read the /etc/hdparm.

confconfiguration file on system

startup. In it is a section for each

hard drive with the following for-

mat:

/dev/sda {

...

}

Modern Linux systems randomly

allocate device names (sda, sdb).

To assign the hdparm settings to a

specific drive permanently, use its

specific UUID:

/dev/disk/by-id/ata-U

SAMSUNG_HD103SJ_S246J1RZB00034 { }

The settings belong between the

curly braces. Each parameter has

its own name. Acoustic manage-

ment is set, for example, to the

value of 128 with the following

command:

acoustic_management = 128

Which name belongs to which hd-

parm parameter is revealed by the

comments at the top of the file.

Conclusions

Hdparm also includes many other

parameters that can be quite dan-

gerous to use. For example, many

SSDs can be protected with apassword, which can lead to data

loss in some situations. Its not a

coincidence that the man page

(man hdparm) warns about these

dangers.

Incidentally, hdparm is only one

useful tool among many; for exam-

ple, the smartmontools can deter-

mine the health status of a hard

drive [2]. n

Info

[1] hdparm: http://hdparm.sourceforge.net

[2] smartmontools: http://sourceforge.

net/apps/trac/smartmontools/wiki




23/24

shop.linuxpromagazine.com/trial

MOBILE USERSsearch for us today at your digital newsstand!

shop.linuxpromagazine.com/trial

Only a swipe away!

Download our convenient

digital editions for your iPad,

iPhone, or Android device.

Visit our apps page for more information: www.medialinx-shop.com/apps

GOOGLE PLAY MAGAZINES

ADMIN Magazine Linux Pro Magazine Ubuntu User Raspberry Pi Geek

APPLE NEWSSTAND

ADMIN Magazine Linux Pro Magazine Ubuntu User Raspberry Pi Geek
http://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/appshttp://www.medialinx-shop.com/apps


24/24

Learn the latest

techniques for better:

network security

system management

troubleshooting

performance tuning

virtualization

cloud computing

on Windows, Linux,Solaris, and popular

varieties of Unix.

Each issue delivers

technical solutionsto the real-world

problems you face

every day.

Real SolutionS

foR Real netwoRkS

FREECD or DVDin Every Issue!

linuxnewmedia.co

m6issuesperyear!
http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/http://www.medialinx-shop.com/

Documents

SysAdmin Day Supplement