Advanced Unix Final Review December 6, 2005. IPSEC

Advanced UnixAdvanced Unix

Final Review

December 6, 2005

IPSECIPSEC

OutlineOutline

• IPsec overview• Alphabet soup being served…

• Security Associations (SA) & SPI’s• Authentication Header (AH) protocol• Encapsulating Security Payload (ESP) protocl• Internet Key Exchange (IKE)• IPsec pitfalls• IPsec vs tunneling (PPTP, L2TP)

IPSec OverviewIPSec Overview

IPSec is a suite of protocols for securing network connections – The details and variations are overwhelming

One cause of the complexity is that IPSec provides a mechanism, not policy– A framework that allows implementation

possible that both ends can agree on

Virtual Private Network (VPN)Virtual Private Network (VPN)

• Secure communications between two hosts or networks

• VPN, is the buzzword that solves all you problems

• IPsec is one of the more popular VPN technology's

What can IPSEC ProvideWhat can IPSEC Provide

• Authentication• Integrity• Access control• Confidentiality• Replay protection (Partial)

Types of VPNsTypes of VPNs

• Host To Host• We’ll do this in class

• Host To Security or Secure Gateway• Secure Gateway To Secure Gateway

• Secure Gateway = Firewall or VPN router• Also referred to as Network To Network

Security Associations (SA)Security Associations (SA)

• A group of security settings related to a specific VPN

• Stored in the SPD (Security Policy Database)• Uniquely Identify IPsec sessions by:

• SPI (Security Parameter Index) a unique number that identifies the session

• The destination IP address• A security protocol or encryption method

• Normally AH or ESP

• A shared secret

Types of IPSEC ConnectionsTypes of IPSEC Connections

• Transport Mode• Does not encrypt the entire packet• Uses original IP Header• Faster

• Tunnel Mode• Encrypts entire packet including IP Header

(ESP)• Creates a new IP header• Slower

IKE (Internet Key Exchange)IKE (Internet Key Exchange)

• UDP port 500• Negotiates connection parameters • ISAKMP (Internet Security

Association and Key Management Protocol)

• Oakley (Diffie-Helmen key exchange)

IPsec PitfallsIPsec Pitfalls

• Complicated• many different ways to configure

• Can be configured insecurely• Client security is an issue• Performance in IPv4 implementation

Advantages of IPSecAdvantages of IPSec

• Encrypts the entire packet, including IP Header (not just layer 4 and higher)

• Can Encrypt any protocol• No Impact on users when using Secure

Gateway to Secure Gateway• Acts independent of IP address

IPsec GuidelinesIPsec Guidelines

• Always use:• 3des or blowfish• SHA1 over SHA and MD5• NEVER USE DES• Tunnel Mode• Main Mode• AH and ESP together• Certificates for production environments

OS Support for IPsecOS Support for IPsec

• OpenBSD, FreeBSD, NetBSD• Linux• Solaris• Windows 2000 (Native)• Windows NT/95/98/Me (Add-on)• Cisco IOS (PIX and Routers)• Others as well....

Squid Proxy ServerSquid Proxy Server

Squid FeaturesSquid Features

Its a caching proxy for:– HTTP, HTTPS (tunnel only)– FTP– Gopher

A full-featured Web proxy cache Designed to run on Unix systems Free, open-source software

Squid SupportsSquid Supports

proxying and caching of HTTP, FTP, and other URLs proxying for SSL cache hierarchies ICP, HTCP, CARP, Cache Digests transparent caching extensive access controls HTTP server acceleration SNMP caching of DNS lookups

Other proxies (besides Squid)Other proxies (besides Squid)

Commercial– Netscape Proxy– Microsoft Proxy Server– NetAppliance’s NetCache (shares some

code history with Squid in the distant past)

– CacheFlow (http://www.cacheflow.com/)– Cisco Cache Engine

What is a proxy?What is a proxy?

Firewall device; internal users communicate with the proxy, which in turn talks to the Internet– Gateway for private address space (RFC 1918)

into publicly routable address space

Allows one to implement policy– Restrict who can access the Internet– Restrict what sites users can access– Provides detailed logs of user activity

What is a caching proxy?What is a caching proxy?

Stores a local copy of objects fetched– Subsequent accesses by other users in the

organization are served from the local cache, rather than the origin server

– Reduces network bandwidth– Users experience faster web access

How proxies workHow proxies work

User configures web browser to use proxy instead of connecting directly to origin servers– Manual configuration for older PC based

browsers, and some UNIX browsers (e.g., Lynx)– Proxy auto-configuration file for Netscape 2.x+ or

Internet Explorer 4.x+• Far more flexible caching policy• Simplifies user configuration, help desk support, etc.

How proxies work (user request)How proxies work (user request)

User requests a page: http://www.rose.edu

Browser forwards request to proxyProxy optionally verifies user’s

identity and checks policy for right to access uniforum.chi.il.us

Assuming right is granted, fetches page and returns it to user

SambaSamba

What is SambaWhat is Samba

Samba is an Open Source/Free Software suite that provides file and print services to SMB clients

Samba current version: 3.20bSamba Home Page:

http://www.samba.org

PrerequisitesPrerequisitesThe following installs:

– Samba– samba-client– samba-common– system-config-samba

samba-swat (optional)

Samba Utilities and DaemonsSamba Utilities and Daemons

net nmbd nmblookup smbclient smbd smbpasswd smbstatus smbtree swat (not part of samba) testparm testprns (deprecated and will be removed in a future Samba release)

Samba users, maps, passwordsSamba users, maps, passwords

Usernames - /etc/samba/smbusersPasswords - /etc/samba/smbpasswdDemo:

/etc/samba/smbusers

Quick StartQuick Start

system-config-samba is used to configure samba server on linux computer

Demo: system-config-sambaSamba usersLinux shares

Sendmail and SMTPSendmail and SMTP

OverviewOverview

Introduction to EmailMessage BreakdownSample MessagesExtensions (MIME)MTA’s and Mailbox Protocols

Email StatisticsEmail Statistics

31 billion emails are sent daily, expected to double by 2006

Email generates about one billion Gigabytes of new “information” per year

Spam accounts for about 40% of all email traffic

http://www.spamfilterreview.com

SMTPSMTP

Originated in 1982 (rfc0821, Jon Postel) Goal: To transfer mail reliably and efficiently

SMTPSMTP SMTP clients and

servers have two main components– User Agents – Prepares

the message, encloses it in an envelope. (Eudora for example)

– Mail Transfer Agent (MTA) – Transfers the mail across the internet

SMTPSMTP SMTP also allows the

use of Relays allowing other MTAs to relay the mail

Mail Gateways are used to relay mail prepared by a protocol other then SMTP and convert it to SMTP

What is Mail?What is Mail? Mail is a text file Envelope –

– sender address– receiver address– other information

Message –– Mail Header –

defines the sender, the receiver, the subject of the message, and some other information

– Mail Body – Contains the actual information in the message

Return-Path: <[email protected]>Delivered-To: [email protected]

Received: by mail.eecis.udel.edu (Postfix, from userid 62)id 17FBD328DE; Wed, 5 Nov 2003 11:27:02

Received: from mail.acad.ece.udel.edu (devil-rays.acad.ece.udel.edu [128.4.60.10])by mail.eecis.udel.edu (Postfix) with ESMTP id

5F41832893 for <[email protected]>; Wed, 5 Nov 2003 11:27:01 Received: by mail.acad.ece.udel.edu (Postfix, from userid 62)id 47509456C; Wed, 5 Nov 2003 11:27:01 Received: from stimpy.eecis.udel.edu(stimpy.eecis.udel.edu [128.4.40.17])by mail.acad.ece.udel.edu (Postfix) with SMTP id 7C2943D79 for <[email protected]>; Wed, 5 Nov 2003 11:26:34 Message-Id: <[email protected]>Date: Wed, 5 Nov 2003 11:26:34 From: [email protected]: undisclosed-recipients: ;

MIME-Version: 1.0

This is a test.

Post Office

Mailbox

Post office

and mail route

Receivers

Mailbox

How SMTP worksHow SMTP works

The Essentials

How about a Demo?

Keyword Arguments

HELO Sender’s Host Domain Name

MAIL FROM: Email Address of sender

RCPT TO: Email of Intended recipient

DATA Body of the message

QUIT

How SMTP worksHow SMTP works

The Extras

Keyword Arguments

RSET

VRFY Name to be verified

NOOP

TURN

EXPN Mailing list to expand

HELP Command Name

Status CodesStatus Codes

The Server responds with a 3 digit code that may be followed by text info– 2## - Success– 3## - Command can be accepted with

more information– 4## - Command was rejected, but error

condition is temporary– 5## - Command rejected, Bad User!

Connection EstablishmentConnection Establishment

TCP Connection Establishment

Message ProgressMessage Progress

Connection TerminationConnection Termination

TCP Connection Termination

Problems with SMTPProblems with SMTP

No security– Authentication– Encryption

Only uses NVT (Network Virtual Terminal) 7-bit ASCII format

E-mails can be forged…..E-mails can be forged…..HELO mail.rose.eduMAIL FROM: [email protected] TO: [email protected]: Dr. Art ZennerTo: Professor RichardsSubject: CIT 2243

Professor Richards,By department decree all students in your CIT 2243

Introduction to Unix class are hereby to be given automatic A’s.Thank you,Dr. Art Zenner.

QUIT

Extensions to SMTPExtensions to SMTP MIME – Multipurpose Internet Mail Extensions

– Transforms non-ASCII data to NVT (Network Virtual Terminal) ASCII data

• Text

• Application

• Image

• Audio

• Video

MIME HeadersMIME Headers

Goes between the Email Header and Body– MIME-Version: 1.1– Content-Type– Content-Transfer-Encoding– Content-Id– Content-Description


Content-Type – Type of data used in the body of the message

Text – plain, unformatted text; HTML

Multipart – Body contains multiple independent parts

Message – The body is whole mail message, part of a message, or a pointer to a message


Image – The message is a stationary image (JPEG or GIF)

Video – The message is an animation (Mpeg)

Audio – The message is 8 kHz standard audio data

Application – The message is a type of data not previously defined


Content-Transfer-Encoding – The method used to encode the messages7 bit – no encoding needed

8 bit – Non-ASCII, short lines

Binary – Non-ASCII, unlimited length lines

Base64 – 6 bit blocks encoded into 8-bit ASCII

Quoted-printable – send non-ASCII characters as 3 ASCII characters, =##, ## is the hex representation of the byte

Base64 EncodingBase64 Encoding

Divides binary data into 24 bit blocks Each block is then divided into 6 bit chunks Each 6-bit section is interpreted as one character

incurs a 25% overhead

11001100 10000001 00111001

110011 001000 000100 111001

01111010 01001001 01000101 00110101

(51) (8) (4) (57)

(z) (I) (E) (5)

Quoted-Printable EncodingQuoted-Printable Encoding

Used when the data has a small non-ASCII portion

Non-ASCII characters are sent as 3 characters First is ‘=‘, second and third are the hex

representation of the byte

01001100 10011101 00111001

00111101 00111001 01000100(=) (9) (D)


Content-Id – Uniquely identifies the whole message in a multiple message

environment Content-Description – defines whether the

body is image, audio, or video

A Multipart, Encoded MIME Message From: [email protected]: [email protected]: Info on Gibson guitarMIME-Version: 1.0Content-Type: multipart/mixed; boundary=17

- 17Content-Type: text/enriched; charset="us-ascii"Content-Transfer-Encoding: 8bitContent-Description: Greetings As promised, I'm getting back to you about the Gibson Southern Jumbo guitar you were Interested in. I've enclosed a spec sheet on the guitar, which is in Microsoft Word.

I guarantee that you'll love it!

- 17Content-Type: application/octet-streamContent-Transfer-Encoding: base64Content-Description: Spec sheet saved as MS Word file

- 17 -

MIME ExampleMIME ExampleDate: Wed, 04 Apr 2001 00:11:37 -0400From: Meghna Naik <[email protected]>MIME-Version: 1.0To: [email protected]: =?gb2312?B?1tDOxA==?= titleContent-Type: text/plain; charset=gb2312Content-Transfer-Encoding: 7bit

a body text, blah, blah

Mail Transfer Agents (MTA)Mail Transfer Agents (MTA)

MTAs do the actual mail transfersMTAs are not meant to be directly

accessed by users.Other MTA’s are:

– Postfix– Qmail– MS Exchange– CC:Mail– Lotus Notes– ….etc.

SendmailSendmail

It's been said that you aren't a real Unix system administrator until you've edited a sendmail.cf file.

It's also been said that you're crazy if you've done so twice.

What is Sendmail?What is Sendmail?

Definition: Sendmail is the most widely used Mail Transport Agent (MTA) on the internet

MTAs send mail from one machine to another. Sendmail is not a client program, which you

use to read your email. Sendmail is one of the behind-the-scenes

programs which move email over the Internet. – Normally it runs as a background daemon– Can even be run out of the super daemon (xinetd)

ImplementationsImplementations

SMTP Gateway– An SMTP gateway allows users on your network

to communicate with others on the Internet without concern as to which local mail software package exists on your network.

– All incoming mail for your network will pass through this gateway which converts the message into the appropriate format specific to your local mail software.

– Similarly, all mail destined for the Internet from your network will pass through this gateway to be sent across the Internet via SMTP

ImplemetationsImplemetations

SMTP Relay ”Warning Will Rogers”– An SMTP relay is a machine that actually

sends the mail across the Internet.– A common misconception is that SMTP

gateways are the same as SMTP relays. This is not always the case.

– There are SMTP gateways that act as relays themselves, but there are also many that do not. If the latter is the case on your network, you'll need to bounce your mail off one of the relays.

Installation MethodsInstallation Methods

RPM installation– Obtained from installation CDs

Binaries (*.tgz)– Obtained from http://www.sendmail.org

Source Code– Obtained from http://www.sendmail.org

The PiecesThe Pieces

The binary: /sbin/sendmail

The configuration file:/etc/mail/sendmail.cf

Supporting files:/etc/mail/access/etc/mail/aliases…and many more

More PiecesMore Pieces

Email messages are stored in the directory:/var/spool/mail – There is a separate file for each user

Email waiting to be sent./var/spool/mqueue

A log of Email sent and received:/var/log/mail

Sendmail Features Sendmail Features

Sendmail uses DNS (Domain Naming System)– But not 100% dependent: [Joe@[192.168.1.1]

DNS provides Mail Exchange (MX) Info Sendmail can do a DNS double-tap

– Lookup up who the client says they are Sendmail default is mail relay off Realtime Blackhole Lists (RBL) Mail Relay checkers - - Open Mail Relay Db

http://www.ordb.org/submit/

http://www.ordb.org/submit/

Sendmail Anti-Spam EnhancementsSendmail Anti-Spam Enhancements

Mailscanner– Minimal anti-spam– Anti-virus integration (scan in/outbound)– http://www.sng.ecs.soton.ac.uk/mailscanner/– Or http://www.mailscanner.info

Spam Assassin– Rule based heuristic – Header and text analysis– Blacklist (RBL)– Vipul's Razor (http://razor.sf.net)– http://www.spamassassin.org

Mail Access ProtocolsMail Access Protocols

The MTAs place the email in the user’s mailbox

The Mail Access Protocols are used by the users to retrieve the email from the mailbox– POP3– IMAP4

All Messages

POP3:

IMAP:Mr Smith

Friends

….headers

Whole message

POP vs. IMAP

Post Office Protocol v3Post Office Protocol v3

SimpleAllows the user to obtain a list of

their EmailsUsers can retrieve their emailsUsers can either delete or keep the

email on their systemMinimizes server resources

Internet Mail Access Protocol v4Internet Mail Access Protocol v4

Has more features then POP3 User can check the email header before

downloading Emails can be accessed from any location Can search the email for a specific string

of characters before downloading User can download parts of an email User can create, delete, or rename

mailboxes on a server

ReferencesReferences

RFC’s: – RFC 821 - Simple Mail Transfer Protocol – RFC0822 - Standard for the Format of ARPA

Internet Text Messages – RFC 1521 - MIME (Multipurpose Internet Mail

Extensions)

E-mail Explained– http://www.sendmail.org/email-explained.html

Sendmail ConfigurationSendmail Configuration

Internal SMTP IssuesInternal SMTP Issues

Vrfy name– Used to verify if a mailbox with the given name

exists in an SMTP server

Expn maillist-name– Used to expand the members of the given

maillist name

Both sources of e-mail address for spammers

Must be disabled

SendmailSendmail

An open source mail transfer agent Original version written by Eric Allman in

1980’s at UC Berkeley Descendant of ARPANET delivermail Very flexible

– Supports different transfer and delivery protocols Very complicated

– Difficult to manage– Configured using sendmail.cf, sendmail.mc

Unfortunately, known for it’s bugs

SendmailSendmail

Security measures:– Sendmail restricted shell: smrsh– Standard security checks– SMTP AUTH– SMTP STARTTLS

Rejecting SPAM– Access database– Anti-spamming relay features– Validating senders

SendmailSendmail

Configuring sendmail– /etc/mail/sendmail.cf

• Actual configuration file

– /etc/mail/sendmail.mc• More user friendly configuration file

– Make sendmail.cf from sendmail.mc• m4 /usr/local/share/sendmail/cf/m4/cf.m4

/etc/mail/sendmail.mc > /etc/mail/sendmail.cf

SendmailSendmail

Turning off exploitable features– Find the line in sendmail.cf that contains

• O PrivacyOptions=

– Add noexpn and novrfy• O PrivacyOptions=noexpn novrfy

– Most strict : goaway– Or set the confPRIVACY_FLAGS in

sendmail.mc• define(`confPRIVACY_FLAGS’, `goaway, noexpn,

novrfy, nobodyreturn’)

SendmailSendmail

SMTP server banner– May give away system info

• 220 192.168.1.1 ESMTP Sendmail 8.10.2+Sun/8.10.2; Tue,14 Jan 2003 09: 28:02-0500 (EST)

– Change SmtpGreetingMessage field in sendmail.cf

SendmailSendmail

Precautions against DoS attacks, in sendmail.mc:– Set confMAX_MESSAGE_SIZE to limit

message size– Set confMAX_DAEMON_CHILDREN to

limit number of processesDoes not prevent DoS attacks

SendmailSendmail

Controlled SMTP relaying in sendmail: FEATURE(access_db)– List the domains you are willing to relay from

in /etc/mail/relay-domains• FEATURE(relay_hosts_only)

– Hosts must also be listed• FEATURE(relay_entire_domain)

– Relay all computers in domain• FEATURE(access_db)

– Enables or disables access database• FEATURE(blacklist_recipients)

– Also look up recipients in access database

SendmailSendmail

Controlled SMTP relaying in sendmail:– List the domains you are willing to relay from in

/etc/mail/relay-domains

• FEATURE(dnsbl)– Use realtime black hole list at mail-abuse.org– 1.5.5.192.blackholes.mail-abuse.org IN A 127.0.0.2

• FEATURE(accept_unqualified_senders)– Allow users without domains

• FEATURE(accept_unresolvable_domains)– Allow users with unresolvable domains

• FEATURE(relay_based_on_MX)– Permit any relay directed to your host

SendmailSendmail

Following features make vulnerable to abuse:– FEATURE(relay_local_from).

• Allows relaying if the message claims to originate at your domain.

– FEATURE(loose_relay_check).• turns off checking for explicit routing

– FEATURE(promiscuous_relay).• turn off all checking for relaying.

SendmailSendmail

Access database– In /etc/mail/access– Allow access by individual domains– Two-tuples : key – action– Key:

• Fully or partly qualified host name• Network or subnetwork address• Specific e-mail addresses• Can also include FROM:, TO:, etc.

SendmailSendmail

Actions:– REJECT

• refuse connections from host

– DISCARD• accept the message but silently discard it, sender will

think message is accepted

– OK• Allow access, overrides other checks

– RELAY• Allow access including relaying

– ERROR:### arbitrary message• Reject mail with customized message

SendmailSendmail

Example– cyberpromo.com REJECT– sendmail.org RELAY– [email protected] ERROR:550 Spammers

do not live here anymore– From:[email protected] REJECT– To:[email protected] REJECT– 193.140 RELAY

Generate database from map– makemap hash /etc/mail/access < /etc/mail/access

Sendmail: smrshSendmail: smrsh

The smrsh program is intended as a replacement for /bin/sh in the program mailer definition of Sendmail.

It's a restricted shell utility that provides the ability to specify, through the /etc/smrsh directory, an explicit list of executable programs available to Sendmail.

smrsh effectively limits Sendmail's scope of program execution to only those programs specified in smrsh's directory.

Sendmail: smrshSendmail: smrsh

The sendmail.cf is configured to run /bin/smrsh by default

To prevent duplicate programs, and do a nice job, it is better to establish links to the allowable programs from /etc/smrsh rather than copy programs to this directory.

For example:

cd /etc/smrsh ln -s /usr/bin/procmail /etc/smrsh/procmail

SendmailSendmail

smsrh:– Form an explicit list of executables that

sendmail is allowed to executesendmail.mc :

– FEATURE(`smsrh’)Advised to be used in all sendmail

versions

SendmailSendmail

Enhanced File Security:– Tight rules for opening files– In general, all read directories should be

owned by root– No .forward in unsafe (group or world

writable) directories

SendmailSendmail

Enhanced File Security:– If too restrictive, set the DontBlameSendmail

option in sendmail.mc– define (`confDONT_BLAME_SENDMAIL’,...)

• ForwardFileInUnsafeDirPath

Allow .forward files in unsafe directories.• ForwardFileInUnsafeDirPathSafe

Allow a .forward file that is in an unsafe directory to include references to program and files.

SendmailSendmail

SMTP-Auth in sendmail:– Install an SASL library

• i.e. Cyrus SASL

– Compile sendmail with right options• APPENDDEF(`confENVDEF', `-DSASL')

APPENDDEF(`conf_sendmail_LIBS', `-lsasl') for Cyrus SASLv1• APPENDDEF(`confENVDEF', `-DSASL=2')

APPENDDEF(`conf_sendmail_LIBS', `-lsasl2') for Cyrus SASLv2to site.config.m4

SendmailSendmail

Set options in sendmail.mc– TRUST_AUTH_MECH(`GSSAPI DIGEST-

MD5')dnl– define(`confAUTH_MECHANISMS', `GSSAPI

DIGEST-MD5')dnl– define(`confDEF_AUTH_INFO',

`/etc/mail/auth/auth-info')dnl– DAEMON_OPTIONS(`a')dnl

Requiring SMTP AUTH– Delete all other means of relaying

SendmailSendmail

To use as client,generate an info file:– client-info: AuthInfo:your.isp.net "U:root"

"P:password“ Generate authentication database:

– # makemap hash client-info < client-info Edit configuration file:

– define(`SMART_HOST',`your.isp.net')– define(`confAUTH_MECHANISMS', `DIGEST-MD5 CRAM-

MD5 LOGIN PLAIN')– FEATURE(`authinfo',`hash /etc/mail/auth/client-info')

SendmailSendmail

SMTP STARTTLS in sendmail– Allow relaying based on certificates– Restrict incoming or outgoing connections

Define following variables:– define(`confCACERT_PATH', `/etc/mail/certs/')– define(`confCACERT',

`/etc/mail/certs/CA.cert.pem')– define(`confSERVER_CERT',

`/etc/mail/certs/my.cert.pem')– define(`confSERVER_KEY',

`/etc/mail/certs/my.key.pem')

SendmailSendmail

${verify} : macro that keeps result of verification– OK : verification succeeded.– NO : no cert presented.– NOT : no cert requested.– FAIL : cert presented but could not be verified,

e.g., the cert of the signing CA is missing.– NONE : STARTTLS has not been performed.– TEMP : temporary error occurred.– PROTOCOL : protocol error occurred (SMTP level).– SOFTWARE : STARTTLS handshake failed.

SendmailSendmail

Relaying based on certificates– If sender not verified, usual relaying– If verified, look up the domain of

certificate issuer, and check access database for that domain

• If result is RELAY, relay• If result is SUBJECT, look up the subject

SendmailSendmail

Example– To allow relaying only for a subset of machines that have a cert

signed by• /C=US/ST=California/O=endmail.org/OU=private/CN=

Darth+20Mail+20+28Cert+29/[email protected]

– use: • CertIssuer:/C=US/ST=California/O=endmail.org/OU=private/

CN=Darth+20Mail+20+28Cert+29/[email protected] SUBJECT

• CertSubject:/C=US/ST=California/O=endmail.org/OU=private/CN= DeathStar/[email protected] RELAY

– Received header• (version=${tls_version} cipher=${cipher} bits=${cipher_bits} verify=$

{verify})

SendmailSendmail

Deciding to continue communication– Two-tuples in access map– Key : clients or servers– Values:

• VERIFY : successful verification required• VERIFY:bits : successful verification

required & cipher bits >= bits• ENCR:bits : cipher bits >= bits

– TLS_Srv, TLS_Clt keywords

SendmailSendmail

Example:– TLS_Srv:secure.example.com ENCR:112 – TLS_Clt:laptop.example.com VERIFY:112

E-mail sent to secure.example.com should be encrypted

E-mail sent from laptop.example.com should be authenticated

SendmailSendmail

Known application bugs and exploits:– CERT advisories, www.cert.org

Do not run sendmail as root– Current versions do not

Sendmail X: new generation of sendmail– Similar to Postfix architecture– Not ready for Prime Time


Apache Web Server

November 29, 2005

Web ServersWeb Servers

Tim Berners-Lee is credited with having created the World Wide Web– he was a researcher at the European High-

Energy Particle Physics lab, the Conseil Européenne pour la Recherche Nucleaire (CERN), in Geneva, Switzerland.

– A tool was needed to enable collaboration between physicists and other researchers


Tim Berners-Lee wrote a proposal called HyperText and CERN in 1989– an extension of the gopher concept but

incorporated many new ideas and features, including:

• HTML (HyperText Markup Language)• HTTP (HyperText Transfer Protocol) • Web browser client software program

– 1989 it was first installed at CERN– 1991 it was fully operational


Main type of web servers exist– For Linux the primary server is Apache

Fedora Core 3 comes with:– Apache– Tux– Stronghold– Zope– BOA– Jigsaw, etc…..

Apache OverviewApache Overview

The “A Patchy” Web server– put together over time by the Apache group– Based on the National Center for

Supercomputing Applications (NCSA) Web daemon.

• The NCSA was created by the National Science Foundation (NSF) and the state of Illinois in 1986 at the University of Illinois

Apache is free, open-source


Configured with Text filesDependableAvailable for numerous platforms,

– even WindowsNetcraft.Com shows 76,000,000 web sites

– 70% are Apache– 21% are Microsoft(http://news.netcraft.com/archives/web_server_survey.html)


There are two core versions of Apache– Version 1.3.x

• Fast enough for most sites• Particularly on 1 and 2 CPU systems

– Version 2.0.x• More features• filters• threads• portability• Scales to much higher loads

Testing ApacheTesting Apache

Now if Apache is runningCreate two files

– index.htm– phptest.php

Save files in:– /var/www/html/– Document Root Directory

Index,htmIndex,htm

Looks like this:

Phptest.phpPhptest.php

File looks like this:


Open the web browser on the system that apache is configured.

In the Address bar type in the IP Address of the system.


Now test Apache from another machine on the network.

Open a web browser then type IP Address in the address bar.

PHPPHP

PHP is a script language for web sites

Comes from PerlGreat for databases and Content

Management Systems (CMS)

PHPPHP

http://<your-ip>/testphp.phpLooks like this:

Apache ConfigurationApache Configuration

Prefork MPMPrefork MPM

Apache 1.3 and Apache 2.0 PreforkEach child handles one connection at a

timeMany childrenHigh memory requirements“You’ll run out of memory before CPU”

Prefork Directives Prefork Directives (Apache 2.0)(Apache 2.0)

StartServersMinSpareServersMaxSpareServersMaxClientsMaxRequestsPerChild

Worker MPMWorker MPM

Apache 2.0 and laterMultithreaded within each childDramatically reduced memory

footprintOnly a few children (fewer than prefork)

Worker DirectivesWorker Directives

MinSpareThreadsMaxSpareThreadsThreadsPerChildMaxClientsMaxRequestsPerChild

KeepAlive RequestsKeepAlive Requests

Persistent connectionsMultiple requests over one TCP socket

Directives:– KeepAlive– MaxKeepAliveRequests– KeepAliveTimeout

Apache 1.3 and 2.0Apache 1.3 and 2.0Performance CharacteristicsPerformance Characteristics

Multi-process,

Multi-threaded,

or Both?

PreforkPrefork

High memory usage Highly tolerant of faulty modules Highly tolerant of crashing children Fast Well-suited for 1 and 2-CPU systems Tried-and-tested model from Apache 1.3 “You’ll run out of memory before CPU.”

WorkerWorker

Low to moderate memory usage Moderately tolerant to faulty modules Faulty threads can affect all threads in

child Highly-scalable Well-suited for multiple processors Requires a mature threading library

(Solaris, AIX, Linux 2.6 and others work well)

Memory is no longer the bottleneck.

Important Performance Important Performance ConsiderationsConsiderationssendfile() supportDNS considerationsstat() callsUnnecessary modules

sendfile() Supportsendfile() Support

No more double-copy Zero-copy* Dramatic improvement for static files Available on

– Linux 2.4.x– Solaris 8+– FreeBSD/NetBSD/OpenBSD– ...

* Zero-copy requires both OS support and NIC driver support.

DNS ConsiderationsDNS Considerations

HostNameLookups– DNS query for each incoming request– Use logresolve instead.

Name-based Allow/Deny clauses– Two DNS queries per request for each

allow/deny clause.

stat() for Symlinksstat() for Symlinks

Options– FollowSymLinks

• Symlinks are trusted.

– SymLinksIfOwnersMatch• Must stat() and lstat() each symlink, yuck!

stat() for .htaccess filesstat() for .htaccess files

AllowOverride– stat() for .htaccess in each path

component of a request– Happens for any AllowOverride– Try to disable or limit to specific sub-

dirs– Avoid use at the DocumentRoot

stat() for Content Negotiationstat() for Content Negotiation

DirectoryIndex– Don’t use wildcards like “index”– Use something like this instead

DirectoryIndex index.html index.php index.shtml

mod_negotiation– Use a type-map instead of MultiViews if

possible

Remove Unused ModulesRemove Unused Modules

Saves Memory– Reduces code and data footprint

Reduces some processing (eg. filters)

Makes calls to fork() faster

Static modules are faster than dynamic

TroubleshootingTroubleshooting

Common pitfalls

and their solutions

Check your error_logCheck your error_log

The first place to lookIncrease the LogLevel if needed

– Make sure to turn it back down (but not off) in production

Check System HealthCheck System Health

vmstat, systat, iostat, mpstat, lockstat, etc...

Check interrupt load– NIC might be overloaded

Are you swapping memory?– A web server should never swap

Check system logs– /var/log/message, /var/log/syslog, etc...

Check Apache HealthCheck Apache Health

server-status– ExtendedStatus (see next slide)

Verify “httpd -V”ps -elf | grep httpd | wc -l

– How many httpd processes are running?

server-status Exampleserver-status Example

Other PossibilitiesOther Possibilities

Set up a staging environment Set up duplicate hardware

Check for known bugs– http://nagoya.apache.org/bugzilla/

Common BottlenecksCommon Bottlenecks

No more File DescriptorsSockets stuck in TIME_WAITHigh Memory Use (swapping)CPU OverloadInterrupt (IRQ) Overload

File DescriptorsFile Descriptors

Symptoms– entry in error_log– new httpd children fail to start– fork() failing across the system

Solutions– Increase system-wide limits– Increase ulimit settings in apachectl

TIME_WAITTIME_WAIT

Symptoms– Unable to accept new connections

– CPU under-utilized, httpd processes sit idle

– Not Swapping

– netstat shows huge numbers of sockets in TIME_WAIT

Many TIME_WAIT are to be expected Only when new connections are failing is it a

problem– Decrease system-wide TCP/IP FIN timeout

Memory Overload, SwappingMemory Overload, Swapping

Symptoms– Ignore system free memory, it is misleading!– Lots of Disk Activity– top/free show high swap usage– Load gradually increasing– ps shows processes blocking on Disk I/O

Solutions– Add more memory– Use less dynamic content, cache as much as possible– Try the Worker MPM

How much free memoryHow much free memorydo I really have?do I really have?Output from top/free is misleading.Kernels use buffersFile I/O uses cachePrograms share memory

– Explicit shared memory– Copy-On-Write after fork()

The only time you can be sure is when it starts swapping.

CPU OverloadCPU Overload

Symptoms– top shows little or no idle CPU time– System is not Swapping– High system load– System feels sluggish– Much of the CPU time is spent in userspace

Solutions– Add another CPU, get a faster machine– Use less dynamic content, cache as much as

possible

Interrupt (IRQ) OverloadInterrupt (IRQ) Overload

Symptoms– Frequent on big machines (8-CPUs and above)– Not Swapping– One or two CPUs are busy, the rest are idle– Low overall system load

Solutions– Add another NIC

• bind it to the first or use two IP addresses in Apache• put NICs on different PCI busses if possible

Virtual HostsVirtual Hosts

Virtual HostingVirtual Hosting

Apache was among the first (the first?) web server to offer Virtual hosting.

With Virtual hosting many URL's can be associated with one IP address– this is useful as IP addresses are a limited

resource.

IIS as supplied free with W2K/XP does not support Virtual Hosting.

Many hosts, one IPMany hosts, one IP

Several Hosts may translate to the same IP address. – IP addresses are a scarce reource.

An Apache server listening on 193.111.200.150 will read the Host: field to see where to look for the page to serve.

Host fieldHost field

http://www.ollieclark.com/acronyms.html

The HTTP request:GET /acronyms.html HTTP/1.1.Host: www.ollieclark.com

Apache users the Host header to see which domain was requested

– this is only available in HTTP/1.1

Apache checks its virtual hosts for the requested Host to see which page to serve or script to run.

An ExampleAn Example

We want to give convenient access to some administrative functions at www.myfirm.co.uk site

We want the URL http://admin.myfirm.co.uk/

to run a script for administering the site. We add a virtual domain admin.myfirm.co.uk

– this is OK as registered .co.uk domain will be myfirm.co.uk.

– In fact 'www' indicates a subdomain

Adding Virtual HostsAdding Virtual Hosts

NameVirtualHost directive specifies an interface on which Apache will accept virtual host requests. – ‘*’ means all interfaces.– can be several

NameVirtualHost directives

– Virtual hosts on the loopback interface

Why?Why?

Why set up virtual hosts on your local computer? Use the Hosts file

– On XP in: C:\WINDOWS\SYSTEM32\DRIVERS\ETC– also on Linux

Add entries:

Then http://admin.myfirm.co.uk/ … will go the local Apache instance which will process the Vhosts as it would in a real set up. Useful for constructing a website locally.

SecuritySecurity

Security – small rantSecurity – small rant

"Security" has three aspects:A. Security. Data is not lost.B. Availability. Data is available to its ownersC. Privacy. Data is not available to others

It is trivial to achieve C on its own. The challenge is to achieve acceptable levels of

A and C while allowing sufficient of B. Advice to keep an Apache web server secure is

often just "Don't allow …".

Access (external)Access (external)

Security as regards visitors to websites hosted by Apache on the web-server.– External security is managed by .htaccess files– and in the main configuration files

An .htaccess file is placed in a directory and manages access to that directory.

.htaccess.htaccess

An .htaccess file may be placed in any directory It controls many features of how Apache treats that directory

– security– execute scripts– use server-side includes

.htaccess files only work if main configuration file has permitted them by an apprpriate AllowOverride directive:

AuthorizationAuthorization

To protect a directory /htdocs/secure we place an .htaccess file in it

This is a text file as above.

Name of the password file

Simplest type of password

Displayed to user

Cannot GET or POST without authorisation

User must give password

Order Allow Deny Order Allow Deny

Three directives really: order, allow, deny Allow directives specify who can access a

resource Deny directives specify who cannot

access a resource Order directive specifies the order in

which the Allow and Deny directives are processed

Order directiveOrder directive

Order directive takes a single argument which is one of:– Deny,Allow– Allow,Deny

Deny,Allow evaluates the Deny directives first and then the Allow directives. So the Allow directives can override the Deny ones. Any request which does not match any directive is allowed. So default is Allow access)

Allow,Deny reverses the ordering. Default is to Deny access.

Allow/Deny directivesAllow/Deny directives

"Allow from" location Recall that the address of the client is supplied location can be a domain name or partial

domain name, an IP address or partial IP address– Allow from comp.leeds.ac.uk would allow

connections originating from within the School– Allow from 129.11 would allow connections fro any

IP address whose first two bytes were 129.11Allow from all is legitimate the Deny directive has the same syntax,

ExampleExample

The example below is one way to allow access to clients in the School of computing

Order Deny,AllowDeny from allAllow from comp.leeds.ac.uk

Order Deny,AllowDeny from allAllow from comp.leeds.ac.uk

Access (internal)Access (internal)

The security situation as regards other users of the web-server.

A web-server has three relevant classes of users:– the administrators (root, wheel)– users (<username>, users)– Apache (nobody, users)

Users manage websites. Apache needs access to the users directories to

retrieve web-pages and execute cgi-scripts

A Typical TaskA Typical Task

We have a script that is going to create and modify the contents of a file.

Visitors to the site will make these modifications

We investigate – file/directory permissions needed to make this

work. – how 'insecure' this leaves the files.

Steps– review file and directory permissions– look at application

PermissionsPermissions

File permissions are 'r', 'w', 'x' set separately for owner, group, and other.– processes run with user and group identity

owner uses owner permissions onlygroup uses group permission onlyother uses other permission only

OctalOctal

Octal digits 0-7 chmod command chmod access file(s) Access can be three octal digits:

– 1st for owner, 2nd for group 3rd for – 4 enables read, 2 enables write, 1 enables execute

So 705 enables rwx for owner, no access group, rx for other, 777 enables everyone rwx, 700 enable rwx for owner but nothing for the group.

PathsPaths

To access a file referenced by a path you must have 'x' permission on every directory on the path.– if 'x' is missing then you cannot list a directory even

To read temp.txt requires 'r' a file

Create and deleteCreate and delete

To create files in a directory a process must have 'w' and 'x' permission on that directory

If you can create a file you can delete any file in the directory– unless the 'sticky bit' is set, then a process can only

delete the files it owns (except the owner of the directory)

ApplicationApplication

Web page visitors.html invites the user to add a comment.

The work is done by visitors.py which opens the file visitors.txt, adds the comment and returns the current contents.

See visitors.html, visitors.py

Sample permissionsSample permissions

Set visitors.py permissions to 755 Set visitors.html to 644 Set visitors.txt to 666 Ser directory of visitors.txt 777 You see these permissions frequently

suggested:– they will work whatever user and group Apache is

running as.– typically Apache runs as user nobody (group

nogroup)

visitors.pyvisitors.py

The script opens visitors.txt for appending. – if the file does not exist it is created– Creation requires write permissions on the

directory Creation permission on the directory

carries with it delete permission– so the script could delete the file if it wanted to.– in fact any Apache script on that server can

delete the file, not just your scripts.

MitigationMitigation

The malicious user needs to know the file system path to the writable directory.

You only need set other permissions for the standard Apache set up. Thus 707, 606, 404 will do.– you can set directory permissions to 705 on your

home directory. Then other users cannot list your directories because they share your group (users, typically)

Some server set ups allow Apache to run as the user who owns the file requested


Linux Kernel

December 1, 2005

Boot ProcessBoot Process

Boot ProcessBoot Process The basic input/output system (BIOS) starts

and checks for hardware devices.– Stored in the computer’s ROM and described as

firmware.– Finds the hardware devices (diskette drives, CD-

ROM drives, and hard drives) needed by the boot process.

– Loads and initiates the boot program stored in the Master Boot Record (MBR, residing in the first sector of the device), and passes control to the boot program.

First Stage Boot LoaderFirst Stage Boot Loader

Two boot loaders are available: Linux Loader (lilo) and Grand Unified Bootloader (grub)

The first-stage boot loader – reads in the partition table and looks for the

second-stage boot loader on the partition configured as bootable (/boot partition).

– Launches the second stage boot loader.

Second Stage Boot LoaderSecond Stage Boot Loader

Presents the user with different OS kernels it has been configured to boot.

Finds the kernel image in the /boot directory.– The kernel binary is named /boot/vmlinuz-<kernel-version>

Places the appropriate initial RAM disk image, called an initrd, into memory. The initrd is used by the kernel to load drivers necessary to boot the system.

Hands control to the kernel.

grub.confgrub.conf

# grub.conf generated by anaconda## Note that you do not have to rerun grub after making changes to this file# NOTICE: You have a /boot partition. This means that# all kernel and initrd paths are relative to /boot/, eg.# root (hd0,1)# kernel /vmlinuz-version ro root=/dev/hdb3# initrd /initrd-version.img#boot=/dev/hdbdefault=0timeout=10splashimage=(hd0,1)/grub/splash.xpm.gztitle Linux Fedora (2.6.5-1.358smp) root (hd0,1) kernel /vmlinuz-2.6.5-1.358smp ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.5-1.358smp.imgtitle Linux Fedora-up (2.6.5-1.358) root (hd0,1) kernel /vmlinuz-2.6.5-1.358 ro root=LABEL=/ rhgb quiet initrd /initrd-2.6.5-1.358.imgtitle Windows 2000 rootnoverify (hd0,0) chainloader +1

Specifies the default boot image willbe the first hard entry

Grub will wait for 10 seconds for input fromthe user before continuing to boot.

The root partition is the second partitionon the first hard drive.

The KernelThe Kernel

Initializes and configures the computer’s memory and configures hardware attached to the system (processors, I/O subsystems, and storage devices).

Decompresses and mounts initrd to load all necessary drivers.

Mounts the root file system in read-only mode and frees any unused memory.

Starts the init process by running /sbin/init.

Initialization ProcessInitialization Process

Init parses the /etc/inittab file to determine the specifics of what programs to run and at what level.– 0 used to halt the system. The system performs an init 0

command and the system is halted.– 1 Puts he system into single-user mode.– 2 Puts the system into a multiuser mode but does not

support networking.– 3 Puts the system into the standard full multiuser mode but

does not automatically start X.– 4 Unused.– 5 X11; Puts the system into standard multiuser mode with a

graphical (X-based) login.

InittabInittab

id:5:initdefault:– Tells the init program what run level to use

after a reboot.si::sysinit:/etc/rc.d/rc.sysinit

– Tells the init program to run the rc.sysinit script.

– Since the second field is empty, the script will run at boot time for all run levels.

rc.systinitrc.systinit

Setting the path and the hostname, and checking whether networking is activated.

Mounting the /proc file system Setting the kernel parameters Setting the system clock Loading keymaps and fonts Starting swapping Initializing the USB controller along with the attached

devices. Checking the root file system. Remounting the root file system as read-write. Loading modules as appropriate.

Inittab (cont’d)Inittab (cont’d)

Starts the /etc/rc.d/rc script with the appropriate run level.– The rc script executes all of the scripts

pointed to by the symblic links contained in the directory for that run level.

– For example, if the run level is 3, the scripts pointed to by the links in /etc/rc.d/rc3.d are run.

/etc/rc.d/rc3.d/etc/rc.d/rc3.dK01yum K35vncserver K74ypserv S12syslog S28autofs S90xfsK05saslauthd K36lisa K74ypxfrd S13irqbalance S40smartd S95anacronK10dc_server K45named K89netplugd S13portmap S44acpid S95atdK10psacct K50netdump K99readahead S14nfslock S55cups S97messagebusK12dc_client K50snmpd K99readahead_early S18rpcgssd S55sshd S97rhnsdK15httpd K50snmptrapd S00microcode_ctl S19rpcidmapd S56rawdevices S99localK20nfs K50tux S05kudzu S19rpcsvcgssd S56xinetd S99mdmonitorK24irda K50vsftpd S06cpuspeed S20random S80sendmail S99mdmpdK25squid K70aep1000 S08iptables S24pcmcia S85gpmK34yppasswdd K70bcm5820 S09isdn S25netfs S87IIimK35smb K74ntpd S10network S26apmd S90crond

All the files here are only symbolic links to the actual scripts that exist in /etc/rc.d/init.d.

The system first runs the scripts whose names start with K to kill the associated processes /etc/rc.d/init.d/<command> stop

The system runs the scripts whose names start with S to start the processes /etc/rc.d/init.d/<command> start

Changing a K name to start with S (e.g., K20nfs S20nfs) makes Linux start the process rather than kill it.

Inittab (cont’d)Inittab (cont’d)

ca::ctrlaltdel:/sbin/shutdown -t3 -r now– Sets the Ctrl+Alt+Delete key combination to indicate a

reboot of the system.– -t option indicates that the init process waits for 3

seconds after sending the warning message and before sending the kill signal.

pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down"

pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Cancelled"

3:2345:respawn:/sbin/mingetty tty3– Initializes the ttys, provides the login and retrieves the user-

input data, and then starts a login process for the user.

Building the KernelBuilding the Kernel

Building the KernelBuilding the Kernel

cd /usr/src/linux-2.4.20-8 make mrproper (optional but recommended) make xconfig

– This command runs a X-based configuration tool that asks you specific question about every kernel configuration.

Building the Kernel (Cont’d)Building the Kernel (Cont’d)

make xconfig, make menuconfig, make config, etc….– Most kernel features have three compilation

options: Y (compiling the option directly into the kernel), N (not compiling the option at all), and M (compiling the option as a kernel module and load it on demand).

– After saving the selection, the configuration file /user/src/linux-2.4.20-8/.config is created.


make dep– Creates dependency information, so that the compiler knows

each component’s dependencies and can compile components as appropriate.

make clean– Cleans up some miscellaneous object files.

make bzImage – Can customize the title in Makefile– Compiles the Linux kernel properly.– The result is a kernel file called bzImage and located in

/user/src/linux-2.4.20-8/arch/i386/boot make modules

– Compiles the kernel modules files


make modules_install– Installs the kernel modules into the directory path

/lib/modules/2.4.20-8/kernel/drivers. make install

– Copies the new kernel and its associated files to /boot directory.

– Builds a new initrd image and adds new entries to the boot loader configuration file.

Use the command ls -l /boot to make sure the initrd-2.4.20-8.img file was created.

Confirm that the file /boot/grub/grub.conf contains a title section with the same version as the kernel package just installed

Documents

Advanced Unix Final Review December 6, 2005. IPSEC