Mail Protocol, Postfix and Mail security · n Postfix n Qmail n MS Exchange n CC:Mail n Lotus Notes...

Mail Protocol, Postfix and Mail security

How Email Appears to Works

2

How Email Appears to Work

How Email Really Works

3

How Email Really Works

Message Format• Envelop

– Routing information for the "postman"

• Message Header – Sender– Recipients (simple, lists, copies, blind copies) – Other fields of control (date, subject)

• Message Body – Free text – Structured document (i.e.: MIME)

Message Format From: GZ Kabir <gzkabir@office.bdcom.com>

To: Diep Kong < diepkong@gmail.com >

Cc: Moin < moin@bofh.im >

Subject: How Internet mail works

Hi Kong,I'm going to be running a course on ...

ü Format was originally defined by RFC 822 in 1982 ü Now superseded by RFC 2822ü Message consists of

ü Header lines ü A blank line ü Body lines

Message Format • Embedded MUA uses inter-process call to send to MTA

• Freestanding MUA uses SMTP to send mail

• Headers added by the MUA before sending

From: GZ Kabir <gzkabir@office.bdcom.com>

To: Diep Kong < diepkong@gmail.com >

Cc: Moin < moin@bofh.im >

Subject: How Internet mail works

Hi Kong,I'm going to be running a course on ...

• Headers added by MTAs

Mail Delivery Agent (MDA) / Mail Transfer Agent (MTA)

From: GZ Kabir <gzkabir@office.bdcom.com> To: Diep Kong < diepkong@gmail.com > Cc: Moin < moin@bofh.im > Subject: How Internet mail works

…..

A message in Transit • A message is transmitted with an envelope:

– MAIL FROM:<gzkabir@office.bdcom.com>RCPT TO:<moin@bofh.im>

• The envelope is separate from the RFC 2822 message • Envelope (RFC 2821) fields need not be the same as the

header (RFC 2822) fields • MTAs are (mainly) concerned with envelopes

– Just like the Post Office... • Error (“bounce”) messages have null senders or Postmaster’s

– MAIL FROM:<>

An SMTP Session telnet mail7i.protonmail.ch 25 220 mail7i.protonmail.ch ESMTP Postfix ...

EHLO mail-pg0-f54.google.com 250- mail7i.protonmail.ch ... 250-SIZE 10485760 250-PIPELINING250 HELP

MAIL FROM:<gzkabir@office.bdcom.com> 250 OK RCPT TO:<moin@bofh.im> 250 Accepted DATA 354 Enter message, ending with “.”

(continued >>>>)

Received: from ...From: ... To: ...etc...250 OK id=10sPdr-00034H-00 quit221 mail-pg0-f54.google.com closing conn... SMTP return codes 2xx OK3xx send more data 4xx temporary failure 5xx permanent failure

DNS Resolution and Transfer Process To find the recipient's IP address and mailbox, the MTA must drill down through the DNS system, which consists of a set of servers distributed across the Internet beginning with the root name servers

root servers refer requests for a given domain to the root nameservers that handle requests for that tld– MTA can bypass this step because it has already knows which domain nameservers handle requests

for these .tlds e.g. bofh.imasks the appropriate DNS server which Mail Exchange (MX) servers have knowledge of the subdomain or local host in the email address

DNS server responds with an MX record: a prioritized list of MX servers for this domain

To the DNS server, the server that accepts messages is an MX server. When is transferring messages, it is called an MTA.

MTA contacts the MX servers on the MX record in order of priority until it finds the designated host for that address domain

sending MTA asks if the host accepts messages for the recipient's username at that domain (i.e., username@domain.tld) and transfers the message

Firewalls, Spam and Virus Filters • An email encountering a firewall may

be tested by spam and virus filters before it is allowed to pass inside the firewall

• filters test to see if the message qualifies as spam or malware

• If the message contains malware, the file is usually quarantined and the sender is notified

• If the message is identified as spam, it will probably be deleted without notifying the sender.

16

Firewalls, Spam and Virus Filters

An email encountering a firewallmay be tested by spam andvirus filters before it is allowedto pass inside the firewall

filters test to see if themessage qualifies as spam ormalware

If the message containsmalware, the file is usuallyquarantined and the sender isnotified

If the message is identified asspam, it will probably bedeleted without notifying thesender.

Troubleshooting Email Issues • transient failures

– If a transient error occurs, the MTA will hang onto the message, periodically retrying the delivery until it either succeeds or fails, or until the MTA decides that the transient issue is really a permanent condition.

• permanent failures – If the MTA cannot deliver the message (it has received a fatal error

message or failed to complete the transfer after repeated attempts), it bounces the message back to the sender. If the sender is a mailing list, the bounce may be handled by automated bounce-handling software.

Concept of Mail Protocols

Component of Email system❏ Mail Transport Agent/Message Transfer Agent (MTA)❏ Mail User Agent (MUA)

❏ Mail Delivery Agent(MDA)

MTAThe actual mail transfer is done through message transfer agents (MTAs). To send mail, a system must have the client MTA, and to receive mail, a system must have a server MTA. The formal protocol that defines the MTA client and server in the Internet is called Simple Mail Transfer Protocol (SMTP)

SMTP uses commands and responses to transfer mail between an MTA client and MTA server.

SMTP Mail transfer Flow

MTA connection setup

MTA Connection Setup..Contd..Sender opens TCP connection with receiver □ Once connected, receiver identifies itself□ 220 service ready□ Sender identifies itself□ HELO <domain>□ Receiver accepts sender’s identification□ 250OK□ If mail service not available, step 2 above becomes: □ 421 service not available

SMTP Exchange of command response pair

Message TransferSender may send one or more messages to receiver

Each message transfer has the following phases:• One MAIL command, identifies originator

•Gives reverse path to use for error reporting•Receiver returns 250 OK or appropriatefail/error message

• One or more RCPT commands, identifies recipients for the message

• Each recipient identified by a separate RCPT• Separate reply for each recipient (250 OK etc.)

• One DATA command transfers message text• End of message indicated by line containing just period (.)

MTA connection termination

MTA connection terminationSender sends QUIT and waits for reply

□ Then initiate TCP close operation

□ Receiver initiates TCP close after sending reply to QUIT

OptimizationIf message is sent to multiple users on a given host,it is sent only once.

❖ Delivery to users handled by destination host

If Multiple messages are ready for given host,a single TCP connection can be used.

❖ Saves overhead of setting up and termination of connection

Possible Errors❖ Host unreachable❖ Host out of operation❖ TCP connection fail during transfer❖ Faulty destination address

User errorTarget user address has changed Redirect if possibleInform user if not

Sender re-queue the mail - will retry till a configurable period of time

SMTP protocol reliability❖ TCP connection is used to transfer mail from sender to receiver over TCP

connection❖ Attempts to provide reliable service❖ No guarantee to recover lost messages❖ No end-to-end ack to sender❖ Error indication report not guaranteed

SMTP receiver❖ Accepts arriving message❖ Places in user mailbox or copies to outbound message queue for forwarding❖ Receiver must

● verify local mail destination● deal with errors❏ Transmission❏ Lack of disk space

SMTP status codes(DSN-Delivery status Notification)Leading digits Indicates catagories

2XX-Positive Completion Reply(Successful)

3XX-Positive Intermediate Reply(Redirection)

4XX-Transient negative completion reply(Client error)

5XX-Permanent negative completion reply(Server error)

SMTP status codes..Cont.

Problems with SMTP

No inherent securityn Authenticationn Encryption

Only uses NVT (Network Virtual Terminal) 7-bit ASCII format

E-mails can be forged…..HELO mail.rose.eduMAIL FROM: carberry@rose.eduRCPT TO: wrichards@rose.eduDATAFrom: Dr. Art ZennerTo: Professor RichardsSubject: CIT 2243

Professor Richards,By department decree all students in your “Introduction to

Unix” class are hereby to be granted automatic A’s.Thank you,Dr. Art Zenner.

QUIT

Extensions to SMTPMIME – Multipurpose Internet Mail Extensionsn Transforms non-ASCII data to NVT (Network Virtual Terminal) ASCII

dataw Text w Applicationw Imagew Audiow Video

MIME and Base64 Encoding

If the internet is the information highway, then the path for email is a narrow tunnel n Only very small vehicles can pass trough

Then how do you send a big truck through a small ravine? n You have to break it down to smaller pieces

and transport the pieces through the ravine, and reassemble the truck

MIME and Base64 Encoding

The same happens when you send a file attachment via email. This is known as encodingn the binary data (256 bits) is transformed to

ASCII text (128 bitsn allowing it to fit through the tunnel

On the recipient's end, the data is decoded and the original file is rebuilt.

Mail Transfer Agents (MTA)MTAs do the actual mail transfersMTAs are not meant to be directly accessed by users.Other MTA’s are:n Postfixn Qmailn MS Exchangen CC:Mailn Lotus Notesn ….etc.

Problems with simple SMTP

The first one relates to message length. Some older implementations cannot handle messages exceeding 64KB.Another problem relates to timeouts. If the Client and server have different timeouts, one of them may give up while the other is still busy, unexpectedly terminating the connection.Infinite mail storms can be triggered. For example, If host 1 holds mailing list A and host 2 holds mailing list B and each list contains an entry for the other one, then a message sent to either list could generate a never ending amount of email traffic unless somebody checks for it.

ESMTP (RFC 2821)

To get around the problems with simple SMTP,extended SMTP has been defined in RFC2821.Clients wanting to use it should send an EHLOmessage instead of HELO initially. If this isrejected, then the server is a regular SMTPserver, and the client should proceed in theusual way. If the EHLO is accepted, then newcommands and parameters are allowed.

POP3Mail access protocol:

POST OFFICE PROTOCOL [RFC 1939]

Simple and limited functionality,Consists of client software and Server software, Server performs user authorization

IMAP: Internet mail access protocol [Version 4]

POP3Retrieves messages from a mail server

Typically, messages are downloaded to your mail client, and deleted from the server

Designed for use with dial-up connections when people were intermittently connected

Listens on Port 110 (with Secure POP generally on port 995)

POP3 Connection Establishment

POP3 protocol sessionroot@amberit ~]# telnet mail.amberit.com.bd 110 Trying 206.71.88.102...Connected to mail.amberit.com.bd. Escape character is '^]'.+OK Dovecot ready. user user@testing.com+OKpass letmein+OK Logged in. list+OK 1 messages:1 482.retr 1+OK 482 octets<snip>Message Headers</snip>This is my short message quit+OK Loggingout.

Basic POP3 commands□ USER <name> - identifies the user□ PASS <password> - authentication for user□ STAT - lists all messages in the mailbox of user□ LIST <msg no.> - lists the content of a message □ RETR <msg no.> - retrieves a particular message □ DELE <msg no.> - Deletes a particular message□ NOOP□ RSET□ QUIT

□ Replies□ +OK□ –ERR

IMAP Protocol session[root@amberit ~]# telnet mail.amberit.com.bd 143 Trying 206.71.88.102...

Connected to mail.amberit.com.bd Escape character is '^]'.* OK Dovecot ready.A1 LOGIN user@testing.comletmein A1 OK Logged in.A2 SELECTInbox* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)*

IMAP Protocol session..cont.OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.* 1 EXISTS* 0 RECENT* OK [UIDVALIDITY 1225333589] UIDs valid* OK [UIDNEXT 2] Predicted next UIDA2 OK [READ-WRITE] Select completed. A3 FETCH 1 BODY[HEADER]* 1 FETCH (BODY[HEADER] {454}<snip> Message Header Delivered</snip> A3 OK Fetch completed.A4 LOGOUT* BYE Logging outA4 OK Logout completed. Connection closed by foreign host.

Webmail

Webmail

Webmail (or web-based email) is any email client implemented as a web application running on a web server

• Online in nature• IMAP connections mostly• Secured HTTP (HTTPS) must• Can be hosted, e.g.: RoundCube, SquirrelMail, Horde, Zimbra, Rainloop, Kite,

MailPile,iRedMail, etc.• Webmail Service providers : Gmail, Outlook, AOL, Yahoo, etc.• Privacy and Security Concern• Easy to configure, easy to host.

Common Threat Landscape• Eavesdropping

• Spamming and Phishing• Spoofing• Malicious Email Attachments

• Replying and Forwarding Issues• CC & BCC Issues

Postfix

Short History• Originally developed in the late 90s at IBM by Wietse

Venema, author of security software (SATAN, TCPwrappers, ...), as ”IBM Secure Mailer”

• Place under an Open Source license, and renamed ”Postfix”

• Intended as a replacement for then insecure mail systems, such as Sendmail

Design Goals• Safety

• Robustness • Performance • Modularity

• Compatibility

Safety• Postfix makes it very hard to lose mails – many checks to

ensure that mail has been written to disk or delivered • Back off mechanisms in case of repeated failure

Security• Collection of daemons working together Doesn't use

environment for communication • Very paranoid about input checking, all allocation is dynamic

(avoiding buffer overflows) • chroot support out of the box for almost all processes &

daemons • No data is ever exchanged directly between processes – all is

done via IPC, and files on disk • Conservative resource usage

Performance • Designed to be fast from the ground up

• Also behaves well with neighbors, doesn't flood them with mail, and instead uses a throughput adaptation

• Will not block delivery for a message if one recipient domain fails

Modular• One program, one function

All programs controlled from ”master.cf” • Many small programs working together, with limited

privileges

• Compatible with Sendmail's /etc/aliases and .forward conventions

Features• Virtual domains – domains and users are completely

independent of system (UNIX) users • Aliases – sendmail compatible

Rewriting – senders, recipients, globallyRBL support (Realtime Blackhole Lists) support Content filtering using pipes, SMTP or milter

• Support for arbitrary mail manipulation with policy services (custom programs talking to postfix)

More Features• Restriction classes Conditional filtering

• Sender or recipient address verification (test email addresses before accepting mail from them)

• TLS support

Core Concepts : maps• In postfix, everything is looked up in a map (table)

• Maps can be in many formats or use many data sources: – hash/btree– regexp/PCRE – CIDR– NIS – LDAP, *SQL (user defined queries)

ArchitectureArchitecture

Basic Postfix Configuration• Two primary configuration files

– main.cf• Main configuration file where all the subsystems are configured (smtp, smtpd,

cleanup, routing, ...)

– master.cf• File controlling how the ”master” process of postfix will launch all the necessary

postfix daemons to perform mail routing, on-demand

Other Configuration Details• Reside in ”maps” mentioned earlier

• Tables containing values and conditions, referred to from main.cf, controlling all aspects such as: – Virtual and local domains – Routing rules– Access control– Rewriting

– ...

Configuration: postconf command• postconf – used to view and edit configuration parameters

• For changing the configuration, it is usually done vi editing ”main.cf” directly

Some Basic main.cfsmtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination

myhostname = group1.group01.net

alias_maps = hash:/etc/aliases

alias_database = hash:/etc/aliases

myorigin = /etc/mailname

mydestination = group1.group01.net, localhost.group01.net, , localhost

relayhost = group1.group01.net

mynetworks = 127.0.0.0/8 61.45.254.0/24 [2001:df0:a:4::]/64 [::ffff:127.0.0.0]/104 [::1]/128

mailbox_size_limit = 0

recipient_delimiter = +

inet_interfaces = all

inet_protocols = all

Some Basic main.cf# TLS parameterssmtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem

smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key

smtpd_use_tls=yes

smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

Some Basic main.cf## in the file /etc/mailname

group1.group01.net

## in the file /etc/aliasespostmaster: rootsysadm: apnic