Design and Management ofEmail Service

張傑生 Jason ChangComputer CenterNational Taiwan University

2005/07/08北區區網訓練課程

jsc@ntu.edu.twhttp://ccms.ntu.edu.tw/~jsc

2

Outline

A brief view of NTU email service Introduction to the architecture and

operation of SMTP Design of a suitable email system

– Webmail solutions

Postfix and simple configuration samples Spam and virus filtering Conclusion

3

Overview

Electronic mail service has already evolved into one of the major Internet applications.

It is not only fundamental, but also a must. Users may become impatient when mails

were delayed, not to mention failed to access their emails.– Imagine we meet the situation of power failure or

cut of water supply

4

A Brief View of NTU Email Service

Current Status– Email statistics

Average daily incoming: 800,000 Truly delivered: 200,000(25%)

– Account statistics Faculty/staff/official use: 4200 Student: 30000 Alumni: 24500

– Account policy Lifelong preserved for retired faculties/staffs and alumni

Server design Incoming relay/filter: 3 Outgoing relay: 1 Webmail: 2 Pop3: 5 for students and 1 for others

5

A Brief View of NTU Email Service

Problems we faced– Account creation and remove– Non-decreasing number of users– Spam

What we have done– Database synchronization with campus mis systems– Single Sign On– Totally encrypted transmission– Spam and virus filtering– Ugly but work architecture for scalability concern

6

Architecture of a Simple Mail System

Consists of the following components– MTA - Mail transfer agent

Sending and forwarding emails Server end

– MDA - Mail delivery agent Delivering emails to recipients’ mailbox Server end

– Pop3/Imap4 Daemons For users to download their mailboxs Server end

– MUA - Mail user agent Reading and composing emails Client end

Architecture of a Simple Mail System

Internet

MTA

Mail Spool

MDA

POP3

MUA

Mail Server

User Client

SMTP

POP3/IMAP4

SMTP

8

Protocols Used for Mail System

Protocols– For computer programs to communicate with each other– Similar to languages that human beings speak

SMTP– Simple Mail Transfer Protocol– Too simple to provide any “advanced features”

Authentication Authorization

POP3– Post Office Protocol version 3– Simple

IMAP4– Internet Message Access Protocol version 4– Fully compatible with internet message standards, e.g. MIME.– Allow messages to be accessed from more than one computer.– Provide support for online, offline, and disconnected modes.– Multiple and share folders.

Mail Forwarding Between Servers

Internet

MTA

Mail Spool

MTA

MDA

POP3

MUA

Mail ServerMail Server

User Client

SMTP SMTP

POP3/IMAP4

10

How to Find the Way to the Destination?

How do we find the way tojsc@ntu.edu.tw?

Internet

MTA

jsc@ntu.edu.tw

?

Mail Server

11

DNS: The Key to All Internet Services

Query DNS server by the address part of email address.(jsc@ntu.edu.tw)

1. Query for MX records

2. Query for A records

3. Give up!

Not required to exist a real machine hosting the address.(ntu.edu.tw)

Adding MX records to improve query performance and to reduce unnecessary query overhead.

12

DNS: Query Results

Precedence value of MX records

– Lower valueswith higher priority

13

DNS: Query Results

When MX record is absent, second query is required to issue.

14

The Use of Mail Relay Gateway

Load sharing and balance Prevent the occurrence of single point of failure Hide your backend mail servers and prevent from

outside attack– Frequently used when you have M$ Exchange servers

Make use of multiple network link connections (different ISPs)

Spam and virus filtering The easiest way for administrators to release current

bottlenecks and to leverage performance. It’s the way how most commercial products work.

The Architecture of Mail Relay Gateways

Internet

Mail Server Relay Server

MTA

Mail Server

MTA

Relay Server

MTA

MTASMTP

SMTP

SMTP

SMTP

LMTP

SMTP

LMTP

16

Planning of a Modern Mail System

This is what system administrators care. Scalability

– User / spool capacity– Virtual user management

Availability– Multiple relay servers– Multiple pop3 servers

Security– Avoid creating system accounts– No access to shell

Performance– Make the right decision

Architecture OS/Software

17

Features Required By Modern Mail Systems

This is what end users care. Ease of use interface

– IMAP4– Webmail

Security– Authentication– Encrypted transmission

Spam and virus filtering– The only effective way to prevent from virus infection is to

remind and force your users to install antivirus softwares.– Stop using M$ systems is alternative solution.

18

Scalability of a Mail System

Traditional way:– Additional servers with different address.

ms1.cc.ntu.edu.tw ms2.cc.ntu.edu.tw ms3.cc.ntu.edu.tw

– Ugly and not professional Modern and Clever way:

– Multiple backend servers– Make use of relay gateways to dispatch and deliver mails– A uniform email address for all of the users.

user@ntu.edu.tw– Virtual user management by central database

A Scalable Mail System Architecture

Internet

Relay Server

MTA

Mail Server 1

MTA

Relay Server

MTA

SMTP

SMTP

SMTP

LMTP

SMTP

LMTP

MTA

MTA

SMTP

LMTP

SMTP

LMTP

Mail Server 2

Mail Server 3

David

Any

Bob

Charlie

20

Improving Scalability and Availability By Adding Multiple POP3 Servers

POP3/IMAP4 accesses are more frequent than incoming mail delivery.

Users may easily aware of the failure of pop3 servers in comparison to smtp/relay servers.(in:out = 20:1)

NFS protocol is used to share the common storage. Mail delivery efforts can be dealt by other servers. Totally rely on the stability of storage server.

– Single point of failure– Could be replaced by high-end NFS servers or NetApp

filers.

Architecture of Multiple POP3 Servers

Mail Spool

POP3

MUA

POP3 2

Bob

POP3/IMAP4

POP3

POP3 3

POP3

POP3 1 Storage

NFS

MUA Alice

POP3/IMAP4

MTA

Mail Server

22

NFS Concerns And Bottlenecks

NFS has its own limitation. Do not expect NFS to scale well. Use gigabit network with high quality switch. UDP is preferred over TCP when network

environment is local and qualified. Consider to apply the Maildir format

– Avoid NFS lock problem Severe enough in Linux and BSD systems Spin lock and dead lock may block further accesses

– Manual interference often requires

23

Make the Right Decision For the Promise of Performance

Capacity planning Reasonable system design and architecture Operating system

– Linux, BSD Always the best and wise choice.

– Solaris It’s been called Slow……laris.

– M$ Are you kidding?

24

Make the Right Decision For the Promise of Performance

MTA– Postfix

Our suggestion!– Sendmail

Not scalable! Lack of new features. Not efficiency in resource management. Rich of documents and tutorials

– Qmail Termination of development.

– Exim Immature.

– M$ Exchange Seeking trouble for yourself?

POP3/IMAP4– UW-IMAP– Courier-IMAP

Supports Maildir format.– Dovecot

Supports both mbox and Maildir format. Still under development.

– Cyrus Proprietary spool format. Not easy to convert between other formats.

25

Webmail: The Trend of Browser Based Mail Reader

Your determination depends on the system architecture. IMAP4 based

– Access the mailbox via IMAP4 protocol locally or remotely.– Flexible, scalable– Speed may be limited by protocol overhead and network transmission.– Horde IMP project

Locally direct access– Access the mailbox locally, proprietary protocol.– Could only run on local mail storage server.– NFS could be applied when deploying multiple servers– Extremely fast!! iff system not overloaded.– Not scale well when you want to expand.– Openwebmail

Excellent project developed by NCKU, Taiwan Marvelous support to Chinese encodings Only support mbox format currently

Webmail Architecture – IMAP4 based

Webmail

Webmail

Browser

HTTP

POP3 1

POP3 2

POP3

Domain A

POP3

Domain B

POP3 / IMAP4

POP3 / IMAP4

POP3 / IMAP4

POP3 / IMAP4

Webmail Architecture – Locally Direct Access

Webmail

Webmail

Browser

HTTP

Mail Spool

POP3

28

Postfix: The Modern and Advance MTA

http://www.postfix.org Developed by Wietse Venema Features

– Easy replacement of existing sendmail system by binaries with same file name and functionality . e.g. sendmail, newaliases

– Human readable configuration file– Multiple small programs with limited execution privilege– Multiple queues to avoid block of normal mails– Backend database lookup table supported– Better support for virtual host– Virtual user system supported– Content filtering interface– Simple spam check/block mechanism

29

Postfix - the Big Picture

http://www.postfix.org/big-picture.html– Yellow ellipsoids are mail programs. – Yellow boxes are mail queues or files. – Blue boxes are lookup tables.

30

Postfix Installation

Debian Linux– apt-get install postfix-tls libsasl7 libsasl-modules-plain

courier-imap

Redhat/Fedora Linux– rpm -ivh postfix-2.2.x.i386.rpm– rpm -ivh cyrus-sasl-2.1.21.i386.rpm

BSD– Use the ports system– /usr/ports/mail/postfix– /usr/ports/security/cyrus-sasl2– /usr/ports/mail/courier-imap

31

Post Installation

Postfix Configuration– master.cf

Similar to inetd.conf Control the behavior of small programs

– In contrast against sendmail, with one binary and one config file

– main.cf The main configuration of the mail system

– In general cases, no modification is required for a simple setup.

Commands to control the postfix program– postfix start– postfix stop– postfix reload

32

Test For Receiving Email

Send an email to yourself. Check system logs

– /var/log/maillog

An simple test from jason@dolphin.cc.ntu.edu.tw to jason@freebsd.csie.nctu.edu.tw

33

Frequently Used Configuration Options In main.cf

myhostname = mail.cc.ntu.edu.tw mydomain = cc.ntu.edu.tw myorigin = $mydomain mydestination = $myhostname,

localhost.$mydomain– On a mail domain gateway,

you should also include $mydomain.

mynetworks_style = host relay_domains = $mydestination

home_mailbox = Maildir/ message_size_limit = 20971520 smtpd_recipient_limit = 1000

34

Configure to Enable SSL

main.cf– smtpd_enforce_tls = yes– smtpd_use_tls = yes – smtpd_tls_cert_file = /usr/local/etc/ssl/smtp.cert– smtpd_tls_key_file = /usr/local/etc/ssl/smtp.key– smtpd_tls_CAfile = /usr/local/etc/ssl/ntuca.crt– smtpd_tls_loglevel = 1 – smtpd_tls_received_header = yes – smtp_enforce_tls = yes– smtp_tls_cert_file = $smtpd_tls_cert_file – smtp_tls_key_file = $smtpd_tls_key_file – smtp_tls_CAfile = $smtpd_tls_CAfile – smtp_tls_loglevel = 1 – smtp_use_tls = yes – smtp_tls_note_starttls_offer = yes – tls_random_exchange_name = /var/run/prng_exch – tls_random_source = dev:/dev/urandom – tls_daemon_random_source = dev:/dev/urandom

master.cf– tlsmgr fifo - - n 300 1 tlsmgr

35

Test for SSL/TLS Connection

telnet to mail server and check for welcome messages.

36

Test for SSL/TLS Connection

Send an email to yourself.– From mx1 to ccms

Check system logs– /var/log/maillog

37

SMTP/Authentication

Traditional mail servers could only determine their relay policy based on connected ip addresses.

Insufficient in face of the modern network– DHCP, NAT– ADSL and dialup users– Roaming users

Mobile, wireless

SMTP Authentication– Username/password authentication for each mail delivery.– Reduce the chance to relay spam mails for anonymous.

38

Setting Up SMTP/Authentication

Install Cyrus-SASL2 library– It could be configured to authenticate via

/etc/passwd, PAM, LDAP, backend databases

Modify /usr/local/lib/sasl2/smtpd.conf– pwcheck_method: saslauthd– mech_list: plain login

Make sure saslauthd is running.

39

Configure to Enable SMTP/Authentication

Since the username/password negotiation is not encrypted. We must configure our server to accept incoming mail authentication only on TLS encrypted connections.

main.cf– smtpd_sasl_auth_enable = yes– smtpd_sasl_security_options = noanonymous– smtpd_tls_auth_only = yes– smtpd_recipient_restrictions = reject_unknown_recipient_domain,

reject_non_fqdn_recipient, permit_sasl_authenticated, reject_unauth_destination

master.cf– smtps inet n - n - - smtpd -o smtpd_tls_wrappermode=yes

The smtpd will listen on port 465 instead of 25.

40

Test For SMTP/Authentication Connection

The username/password is required to be encoded in BASE64. The Perl MIME::Base64 module could be installed.

– /usr/ports/converters/p5-MIME-Base64 perl -MMIME::Base64 -e 'print encode_base64("user\0user\0passwd");‘

– dGVzdDEAdGVzdDEAcGFzczE=

41

Test For SMTP/Authentication Connection

Send an email to yourself. Check system logs

– /var/log/maillog

42

SMTP Transaction Example

43

Email Headers

Envelope header– Negotiated between mail servers– Typical tricks that SPAM mail plays

Why do I receive mails that are not destined to me? Content Text

– Mail servers will not examine the content in general cases.

44

Real World Snail Mail Example

Postman could only deliver the mail by envelope information.

Sender address is not verified when email delivery.

– Trick of SPAM mails. Content may be

irrelevant or conflict with envelope.

45

Postfix: Spam Filtering Functions

SMTP access restriction– Refer to the previous SMTP transaction example– http://www.postfix.org/uce.html

Restriction list name Effect of REJECT or DEFER result

smtpd_client_restrictions Reject all client commands

smtpd_helo_restrictions Reject HELO/EHLO information

smtpd_sender_restrictions Reject MAIL FROM information

smtpd_recipient_restrictions Reject RCPT TO information

46

Postfix: Spam Filtering Functions

Access maps– Similar to sendmail access map– Control relay permission based on IP addresses– main.cf

smtpd_client_restrictions = check_client_access hash:/usr/local/etc/postfix/access– Execute postmap access after update

Dynamic IP ranges should be blocked– http://www.twia.org.tw/SPAM-930412(news).htm

47

Postfix: Spam Filtering Functions

Realtime black list– Use with caution

The definition and meaning of each rbl Too many TW hosts are on the lists

– main.cf maps_rbl_domains = bl.spamcop.net, relays.ordb.org,

orbs.dorkslayers.com smtpd_client_restrictions = reject_maps_rbl

48

Postfix: Spam Filtering Functions

SMTP access restriction– smtpd_client_restrictions

maps_rbl_domains = bl.spamcop.net, relays.ordb.org, orbs.dorkslayers.com

smtpd_client_restrictions = reject_rbl_client relays.ordb.org reject_rbl_client bl.spamcop.net reject_maps_rbl check_client_access regexp:/usr/local/etc/postfix/access reject_unauth_pipelining reject_unknown_client (use with caution)

– smtpd_helo_restrictions smtpd_helo_required = yes smtpd_helo_restrictions =

reject_invalid_hostname reject_unknown_hostname reject_non_fqdn_hostname

49

Sample Rejected Logs

50

Postfix: Spam Filtering Functions

SMTP access restriction– smtpd_sender_restrictions

smtpd_sender_restrictions = reject_unknown_sender_domain reject_non_fqdn_sender check_sender_access hash:/etc/postfix/sender-access

– Useful to block SPAM mails that fake envelope mail from header

51

Postfix: Spam Filtering Functions

SMTP access restriction– smtpd_recipient_restrictions

smtpd_recipient_restrictions = check_recipient_access regexp:/usr/local/etc/postfix/recp-access reject_unknown_recipient_domain reject_unauth_destination permit_sasl_authenticated permit_mx_backup

52

Postfix: Spam Filtering Functions

Header check– Both envelope and content headers– main.cf

header_checks = regexp:/usr/local/etc/postfix/header

53

Postfix: Spam Filtering Functions

Simple content check– main.cf

body_checks = regexp:/usr/local/etc/postfix/body

Content filtering interface– http://www.postfix.org/FILTER_README.html– Amavisd-new as an example

54

Virus and Spam Mail

Virus mail– Contain virus infected attachment– How do we determine if the mail is virus infected?

Simple and easy. Base on the inspection and judgment of antivirus software.

Spam mail– UCE/UBE

Unsolicited Commercial Email Unsolicited Bulk Email

– How do we determine if the mail is a Spam mail? Difficult. It’s not a yes/no problem. One person’s meat may be another’s poison.

55

Issues When Dealing With SPAM

Do not block or discard users’ mail unless you have the authorization and delegation.

The only safe and acceptable assistant you can provide is to score and tag each mail and let your users to do the rest themselves.

To warn or not to warn the senders/recipients is a dilemma!

– Virus and spam senders nowadays always fake the from address.

– System notification mails may be another kind of SPAM. The result will be blamed by your users and blocked by other mail servers.

56

Greylisting

Recently proposed (2003) method to block spam and virus mails.

RFC requires MTA to be capable to support queue and retry of temporarily undeliverable mails.

– Spam and virus always fire and forget.– For efficiency concern, usually they do not retry.

How it works?– In brief, block every delivery at its first try.– For each mail delivery, check if the the triplet(source ip,

sender, recipient) was seen recently. If yes, pass the mail. If not, reject the delivery with “450 Please retry later.”

57

Greylisting

Drawback– Mail delivery will suffer a delay ranging from several

minutes to hours. Depending on the remote MTA configuration.

Risk– Sites that do not retry their mail delivery.

Some bank notifications were observed to try only once. Mail sending programs that process the SMTP transaction

themselves without calling the “sendmail” MTA interface.

Suggestion– Whitelisted known hosts to prevent from delay and loss of

mail.

58

Greylisting

Benefit– Effectively reduce the volume of spam and virus.– Effectively reduce both cpu and disk load of

servers.

59

Softwares To Block/Filter Spam and Virus

Postfix– Content filter interface

Postgrey– Greylisting policy server

Amavisd-new– http://www.ijs.si/software/amavisd– http://www.ijs.si/software/amavisd/README.postfix

Clam Antivirus– http://www.clamav.net– A GPL virus scanner

SpamAssassin– http://spamassassin.apache.org

60

Software Architecture for Filtering

Internet

Postfix

Amavisd

Final Destination

Mail Server

SMTP

SpamAssassin

Antivirus

1

2

3

4

5

6

Postfix

Storage

Local Delivery

Amavisd and

SpamAssasin Host

Antivirus

Scanning Host

Incoming Mail

Gateway

Performing

Greylisting and

Fundamental

Spam Check

61

Amavisd-new

A high performance interface between MTA and content checkers.

– Calling external antivirus programs to do virus scanning.– Calling external spamassassin program to do spam level

determination.– CPU intensive workloads.– Can be flexibly configured to pass, discard, or quarantine

mails based on user defined policy. Pass spam mails with score > 10 with subject prepended the

*** SPAM *** keyword. Quarantine spam mails with score > 20. Discard spam mails with score > 30. Quarantine virus mails.

62

Spamassassin

Spam level scoring software. Rich set of tests to identify various spam signatures.

– Keywords, bad headers, encodings

Use bayesian analysis to help scoring.– Training the bayesian database using know spam and ham

mails.– Default to enable the auto-learn feature.

Calling external programs to check if the mail was a known spam.

– Use hash of mail content as the query key.– Razor, DCC, Pyzor.

63

Spamassassin

RBL(realtime black list) look up based on sender ip address.– RBL may contains too many ill-administrated sites.– Use the result as an addition of spam score.– Do not block remote sites depend solely on RBL.

SURBL(Spam URI realtime black list) look up based on the URIs within the content of mail.– Spammers may keep changing their sending IP

addresses.– The URIs in the content may be the final

destination the advertisement want people to visit.

64

Postfix: Content Filter Configuration

master.cf– smtp-amavis unix - - y/n - 2 smtp

-o smtp_data_done_timeout=1200 -o smtp_send_xforward_command=yes -o disable_dns_lookups=yes

– 127.0.0.1:10025 inet n - y/n - - smtpd -o content_filter= -o local_recipient_maps= -o relay_recipient_maps= -o smtpd_restriction_classes= -o smtpd_client_restrictions= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o mynetworks=127.0.0.0/8 -o strict_rfc821_envelopes=yes -o smtpd_error_sleep_time=0 -o smtpd_soft_error_limit=1001 -o smtpd_hard_error_limit=1000

main.cf– content_filter = smtp-amavis:127.0.0.1:10024

65

Amavisd-new Configuration

amavisd.conf– $max_servers = 30; # number of pre-forked children– @av_scanners = ( ….– $final_virus_destiny = D_DISCARD; # (defaults to D_BOUNCE) – $final_banned_destiny = D_BOUNCE; # (defaults to D_BOUNCE) – $final_spam_destiny = D_DISCARD; # (defaults to D_REJECT) – $final_bad_header_destiny = D_PASS; # (defaults to D_PASS),

D_BOUNCE suggested– $QUARANTINEDIR = '/var/virusmails/infected';– $sa_tag_level_deflt = 1; # add spam info headers if at, or above that level– $sa_tag2_level_deflt = 9; # add 'spam detected' headers at that level– $sa_kill_level_deflt = 20; # triggers spam evasive actions– $sa_dsn_cutoff_level = 20; # spam level beyond which a DSN is not sent– $sa_quarantine_cutoff_level = 30;

Raise the tag2 value to avoid removing users’ mail.

66

SpamAssassin Configuration

Built-in tests– http://spamassassin.apache.org/tests.html

local.cf– ok_languages en ja zh– ok_locales en ja zh– score SUBJ_ILLEGAL_CHARS 0– score FROM_ILLEGAL_CHARS 0– score HEAD_ILLEGAL_CHARS 0– score CHARSET_FARAWAY 1.0– score CHARSET_FARAWAY_HEADER 1.0– score MIME_CHARSET_FARAWAY 1.0

– header NTU_SMTP Received =~ /mx3.cc.ntu.edu.tw \(relay5.tp1rc.edu.tw \[163.28.16.251\]\) \(using TLSv1 with cipher DHE-RSA-AES256-SHA \(256\/256 bits\)\) .* by mx1.cc.ntu.edu.tw \(Postfix\)/

– score NTU_SMTP -15.0– describe NTU_SMTP mx3.cc.ntu.edu.tw

– header HINET_MSR Received =~ /msr\d{1,2}.hinet.net \(msr\d{1,2}.hinet.net \[168.95.4.1\d{1,2}\]\) by mx1.cc.ntu.edu.tw \(Postfix\)/

– score HINET_MSR -10.0– describe HINET_MSR msr.hinet.net

67

Sample Log of Postfix

Each incoming mail may have up to two transaction logs.

68

Sample Log of Amavisd

69

Sample Headers of User’s Mail

70

Sample Headers of User’s Mail

71

Sample Headers of User’s Mail

72

Sample Screenshots of Outlook Express

73

Make Use of the Filter Rules on your MUA

Outlook Express– Can only filter based on limited headers.

Subject, Sender, etc. Configure to move mails having subjects beginning with

*** SPAM *** to an SPAM folder.

Outlook, Netscape, Thunderbird– You can filter mails based on any headers.– Determine your own spam level threshold

referring the X-Spam-Level: ******

74

Conclusion

Make use of public domain, open source software to construct your own mail system.

Official web sites, mailing lists, and google are always your best friends.

Feedback and contribute as possible as you can.