44
IBM Research © 2007 IBM Corporation Postfix retrospective Wietse Venema IBM T. J. Watson Research Center Hawthorne, NY, USA

Postfix retrospective

  • Upload
    erica

  • View
    44

  • Download
    0

Embed Size (px)

DESCRIPTION

Postfix retrospective. Wietse Venema IBM T. J. Watson Research Center Hawthorne, NY, USA. Postfix expectations before the first release. - PowerPoint PPT Presentation

Citation preview

Page 1: Postfix retrospective

IBM Research

© 2007 IBM Corporation

Postfix retrospective

Wietse VenemaIBM T. J. Watson Research CenterHawthorne, NY, USA

Page 2: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix expectations before the first release

[Postfix]: No experience yet, but I’d guess something like a wisened old man sitting on the porch outside the postoffice. Looks at everyone who passes by with deep suspicion, but turns out to be friendly and helpful once he realises you’re not there to rob the place.

Article in alt.sysadmin.recovery, 1997

See http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html for contemporary comments on other mail systems.

Page 3: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Overview

Good PR can have unanticipated impact.

Why write yet another UNIX mail system?

Postfix architecture and inspiration.

Adding antispam/virus support.

Mission accomplished.

Market share.

Lessons learned.

Page 4: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Good PR can have unanticipated impact

Page 5: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

New York Times, December 1998

SHARING SOFTWARE, IBM TO RELEASE MAIL PROGRAM BLUEPRINT

By JOHN MARKOFF

- - -

The program, Secure Mailer, serves as an electronic post office for server computers connected to the Internet. It was developed by Wietse Venema, an IBM researcher and computer security specialist.

- - -

Currently about 70 percent of all e-mail worldwide is handled by Sendmail, a program that has been developed over more. . .

Page 6: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix (Secure Mailer) project

Business model: if it’s good for the Internet, then it’s good for e-business, and therefore it’s good for IBM.

One year after the first public release, several news articles began to mention Postfix as the project that triggered IBM’s adoption of open source.

Reportedly, this started when IBM’s top management saw the NY Times article.

Page 7: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

How Postfix (Secure Mailer) helped IBM to embrace Open Source + Linux

Page 8: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Building up momentum

June 1998 IBM joins the open source Apache project.

Sept 1998 JIKES Java compiler open source release.

Sept 1998 PKIX public key infrastructure software open source release under the name “Jonah”.

Dec 1998 Secure Mailer open source release under the name “Postfix”. IBM’s CEO starts asking questions.

1999 IBM develops an open source license (now CPL), and announces support for Linux on all platforms.

Page 9: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Why (not) write yet another UNIX mail system

Page 10: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

New code, new opportunities for bugs(creating bugs faster than they can be found and eliminated)

Source code line counts for contemporary software:

Windows/XP: 40 million; Vista 50+ million

Debian 2.2: 56 million; 3.1: 200+ million

Wietse’s pre-Postfix average: 1 bug / 1000 lines1.

Postfix initial release: 30k lines of opportunity1,2.

1Not included: comment lines, or bugs found in development.

2Today: 97k lines of code (194k with comments + documentation).

Page 11: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

CERT/CC UNIX mail advisories(it’s not just about Sendmail)

Bulletin Software Impact CA-1988-01 Sendmail 5.58 run any command CA-1990-01 SUN Sendmail unknown CA-1991-01 SUN /bin/mail root shell CA-1991-13 Ultrix /bin/mail root shell CA-1993-15 SUN Sendmail write any file CA-1993-16 Sendmail 8.6.3 run any command CA-1994-12 Sendmail 8.6.7 root shell, r/w any file CA-1995-02 /bin/mail write any file

Page 12: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

CERT/CC UNIX mail advisories

Bulletin Software Impact CA-1995-05 Sendmail 8.6.9 any command, any file CA-1995-08 Sendmail V5 any command, any file CA-1995-11 SUN Sendmail root shell CA-1996-04 Sendmail 8.7.3 root shell CA-1996-20 Sendmail 8.7.5 root shell, default uid CA-1996-24 Sendmail 8.8.2 root shell CA-1996-25 Sendmail 8.8.3 group id CA-1997-05 Sendmail 8.8.4 root shell

Page 13: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix primary goals(It’s not just about security)

Compatibility: make transition easy.

Wide deployment by giving it away.

Performance: faster than the competition.

Security: no root shells for random strangers.

Flexibility: C is not an acceptable scripting language.

Reliability: behave rationally under stress.

Easy to configure: simple things should be easy.

Page 14: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix architecture and inspiration

Page 15: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Traditional BSD UNIX mail delivery architecture(impersonation requires privileges; monolithic model hinders damage control)

mailbox file

Sendmail*

/bin/mail*

to networkfrom network

local submission

local delivery

* uses root privileges

to |command**

to /file/name**

** in per-user .forward files and in per-system aliases database

owned by recipient

executed as recipient

Page 16: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix client-server service-oriented architecture(omitted: non-daemon programs for submission and queue management)

smtpd

localpickup

smtpdinternetsmtp

server

otherdaemons

smtpdsmtpd

localdelivery

smtpdsmtpdsmtpclient

mail storeinternetetc

mailbox|command/file/name

mailqueue

privileged

smtpdsmtpdto externaltransports

uucpfaxpager

privileged

unprivileged

unprivileged

unprivileged

unprivileged

smtp/lmtpclient

(local submission)

= root privilege= postfix privilege

input interfaces core output interfaces

Page 17: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Major influences on Postfix architecture

TIS Firewall smap/smapd: least privilege, chroot jail, “air gap” between receiving and delivering processes.

qmail: parallel deliveries; the maildir format (the MH mail handling system introduced a “one file per message” mailbox store 20 years before qmail).

Apache: reuse processes multiple times.

Sendmail: user interface; lookup table interface.

Traditional routers: multiple interfaces/encapsulations, central core, but alas no queue-skipping fast path :-(

Page 18: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Adding anti-spam/virus support, part 1: Use standard protocols whenever you can.

“Junk mail is war. RFCs do not apply.”

Wietse on Postfix mailing list, 2001

Page 19: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

1999 - Melissa ravages the Internet

You can run from Windows but you can’t hide: Postfix becomes deeply involved with malware distribution.

Short term: block “known to be bad” strings in message header text (body strings come later).

Long-term: delegate deep inspection to third-party software.

Emergence of specialized protocols: CVP, Milter, etc.

We already use SMTP for email distribution. Why can’t we also use SMTP to plug in anti-{spam,virus}?

Page 20: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix content inspection via SMTP (post queue)

Red = dirty, green = clean.

But it can’t be that simple, right?

Using two MTAs must be wasteful!

MTA 1 Filter MTA 2in outsmtp smtp

Page 21: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix content inspection via SMTP (post queue)

Folding the two MTAs on top each other saves some resources, but also increases complexity.

networksmtp

server

mail queue

smtp client

smtp server

smtp client

content filter

local delivery

local pickup

mailbox command file

network

local submit

MTA 1 = MTA 2

Page 22: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix post-queue anti-spam/virus support

The advantages of post-queue SMTP-based anti-spam/virus filters outweigh the disadvantages:

– Compatibility: many products are SMTP enabled. SMTP is well understood, as are the workarounds for common implementation errors.

– Performance: decoupling the remote network latencies from local filter concurrencies allows for better resource management than possible with e.g. Milters.

Workarounds for loss of original SMTP client context:

– Xforward, etc.

Page 23: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Adding anti-spam/virus support part 2: Embrace de-facto standards.

“It's not the spammers who destroy [email], it's those who insist on broken anti-spam measures.”

Wietse on Postfix mailing list, 2003

Page 24: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

2005 - Proliferation of authentication technologies

SenderID, Domainkeys, DKIM, SPF, BATV, SRS, and the end is not in sight.

Problem: using SMTP-based filters just to “stamp” or “verify” can be clumsy (e.g., missing original SMTP client context). Tighter coupling to MTA is desirable.

Building into the MTA is not practical; besides, many (Linux) distributions are two years behind on Postfix.

Solution: adopt Sendmail Milter protocol and open up access to a large collection of available applications.

Page 25: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Retrofitting Milter support into a distributed MTA

Red = dirty, green = clean.

The effort was heroic, but the reward was sweet.

network

local pickup

queue inject

milter application(s)

smtp server

local submit

mail queue

Postfix (subset)

smtp events

header body...

Page 26: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix author receives Sendmail innovation award

MOUNTAIN VIEW, Calif. October 25th, 2006 Today at its 25

Years of Internet Mail celebration event, taking place at the

Computer History Museum in Mountain View, California, Sendmail,

Inc., the leading global provider of trusted messaging, announced

the recipients of its inaugural Innovation Awards.

. . .

Wietse Venema, author, for his contribution of extending Milter

functionality to the Postfix MTA.

http://www.sendmail.com/pdfs/pressreleases/Sendmail%20Innovation%20Awards_10%2025%2006_FINAL.pdf

Page 27: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Mission accomplished

Page 28: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Catching up on Sendmail

Initial trigger: the Postfix 2.2 source tar/zip file was larger than the Sendmail 8.13 source tar/zip file.

Analyze eight years of Sendmail, Postfix, and qmail source code:

– Strip comments (shrinking Postfix by 45% :-).

– Format into the “Kernighan and Ritchie C” coding style (expanding qmail by 25% :-).

– Delete repeating (empty) lines.

Page 29: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

MTA Source lines versus time

Page 30: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Benefits of Postfix partitioned security architecture

Normally, adding code to an already complex system makes it even more complex.

– New code has unexpected interactions with already existing code, thus reducing over-all system integrity.

The Postfix architecture encourages separation of functions into different, untrusting, processes.

– Each new major Postfix feature is implemented as a separate server with its own simple protocol.

– This separation minimizes interactions with already existing code, thus preserving system integrity.

Page 31: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Mission accomplished

Postfix 2.3 (now 2.4) is complete enough that I am no longer embarassed to recommend it to other people.

– Built-in: TLS, SASL, MIME, IPv6, LDAP, SQL, DSN.

Further extension via plug-in interfaces:

– DomainKeys, DKIM, SenderID, SPF.

– Non-Cyrus SASL authentication, content inspection.

– Sendmail Milter applications, SMTP server access policy.

Todo: clean up internals, hard-coded behavior, etc.

Page 32: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Market share (lies, d*mned lies, and ...)

Page 33: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Interesting result, but what does it mean?Query = sendmail, postfix, exim, qmail

Rel

ativ

e se

arch

vol

ume

Page 34: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Introducing Google trends

Website: trends.google.com.

Search for RELATIVE popularity of search terms in Google queries (recursive Google?).

Result is a time distribution, with each popularity displayed in its own color.

Peaks are correlated with on-line news articles.

Page 35: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Pollution by common words and name collisionsQuery = postfix, prefix, infix

Rel

ativ

e se

arch

vol

ume

Page 36: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Tweaking the query to avoid biasQuery = sendmail server, postfix server

Rel

ativ

e se

arch

vol

ume

Page 37: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Google trends caveats

As always, the answer you get is only as good as the question you ask. Beware of name collisions, common words, and other forms of pollution.

Regardless, one thing is clear: only a minority of Google search users are interested in mail server technology, and their proportion is steadily declining.

Page 38: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Fingerprinting 400,000 company domains remotely

After: Ken Simpson and Stas Bekman, O’Reilly SysAdmin, January 2007.

http://www.oreillynet.com/pub/a/sysadmin/2007/01/05/fingerprinting-mail-servers.html

unknown: 15% other: 20%

Page 39: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix lessons learned

Good PR does make a difference. It’s easy to under-estimate how swiftly a large company can move.

Don’t re-invent mechanisms that already work (e.g., SMTP, Milter, maildir, lookup tables). Invent sparingly.

Build the basic stable protols into the MTA: SMTP, LMTP, TLS, SASL, IPv6, DSN, MIME, LDAP, SQL.

Use plug-ins for future proofing: Anti-Spam, Anti-Virus, DKIM, SenderID, SPF, greylist, etc.

Know when to stop, at least for a while.

Page 40: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix Pointers

The Postfix website at http://www.postfix.org/

Books by other people:

– Ralf Hildebrandt, Patrick Koetter, The Book of Postfix (2005).

– Kyle Dent, Postfix The Definitive Guide (2003).

– Richard Blum, Postfix (2001).

– Original books and translations in German, Japanese, Chinese, Czech, and other languages.

Page 41: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Backup

Page 42: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Where did all that code go?(from Postfix alpha to Postfix 2.3)

4x Growth in size, 8400 lines/year, mostly same author.

Small increase:

– 1.3x Average program size (800 to 1100 lines).

Medium increase:

– 2.5x Program count (from 15 to 36).

Large increase:

– 4x Library code (from 13000 to 52000 lines).

No increase: number of privileged programs.

Page 43: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix content inspection via SMTP (pre-queue)

SMTP “pass-through” hack built into SMTP server.

No decoupling of remote network latencies from local filter concurrencies.

Less scalable, due to poorer resource management.

But the user wanted pre-queue spam/virus filtering.

smtp server

mail queue

content filter

smtp server

Postfix (subset)

Page 44: Postfix retrospective

IBM Research

© 2007 IBM Corporation Postfix retrospective - Wietse Venema - CEAS 2007

Postfix RFC lines versus time