Article: Why Internet Mail Gets Out of Control and How to Avoid Going Postal

This article is the first in a series of three articles on Internet e-mail. In the first installment I’ll explain the central issues surrounding capacity planning for Internet e-mail and give practical advice, pointers and potential pitfalls, for technology planners. I have also included an example capacity model for network traffic projections.

Capacity planning has often been viewed as more of an art than a science. Examples of failure abound ranging from severe inadequacies leading to catastrophic system failures to grossly over-engineered systems that waste millions dollars. Internet e-mail is no exception. In fact, recent years have seen a radical increase in Internet mail and most companies have had difficulty keeping up.

Most companies start out thinking of Internet e-mail as a simple gateway service for a few users to communicate with a limited number of business partners and clients. This way of thinking stems from legacy e-mail gateway services like MCI Mail, Compuserve Mail, SprintMail and AT&T Mail that provided business-to-business e-mail communications through proprietary infrastructures at a relatively high cost compared to Internet e-mail.

Within a few months of installing or upgrading Internet e-mail facilities, many companies discover that they haven’t invested enough in their SMTP infrastructure and, as their systems become less reliable and begin to fail, they feel the pain of making do with inadequate resources. The resources are of course, hardware, software, and bandwidth.

Top 10 Planning Issues for Internet Mail

In the last few years, since approximately 1995, most companies have seen a dramatic and continuing increase in Internet e-mail traffic. Planners have often been caught off guard by increases of 500% or 1000% inside of 24 months. There are many reasons for this but here are the top ten:

1. CHANGING ROLE OF SMTP: The role of SMTP at tends to change from a simple gateway service to an enterprise e-mail hub linking business units, clients, and business partners. To understand a company’s growth pattern, it is necessary to understand its business, particularly in terms of potential mergers and acquisitions, or other potential forms of rapid growth. Technology planners must be able to map business developments to technology requirements. For most companies, this means that business and technology planners need to work more closely together. As the economies become increasingly Internet-centric, this will become the norm rather than the exception.

2. MISSION-CRITICAL E-BUSINESS: Internet e-mail is a key component in electronic commerce. This often means that not only will the volume of Internet e-mail go up but the importance of Internet e-mail increases. When a business stands to loose revenue because of interruptions of Internet e-mail it’s time to rethink the SMTP infrastructure.

3. RAPID ADOPTION OF STANDARDS: The messaging industry and business at large has selected Internet standards for messaging making the SMTP, MIME, POP3, IMAP4, LDAP, S/MIME, X.509 and SSL standards increasingly important. This means that companies are implementing systems supporting these technologies at a record pace.

4. BUSINESS-TO-BUSINESS INTERNET: Traditional proprietary business-to-business e-mail communications (MCI Mail, Compuserve Mail, SprintMail, AT&T Mail) are being dropped at a rapid pace in favor of Internet-based communications. It is worth noting that the overall volume of business-to-business e-mail tends to increase and this trend is likely to continue.

5. THIRST FOR BANDWIDTH: Often the network infrastructure for SMTP, meaning (1) server backbones, (2) internal WANs, and (3) connection(s) to the Internet, turn out to be unable to handle the load. Companies do not always realize the interdependency of these three aspects of networking and their impact on Internet e-mail reliability.

6. COMPETITION FOR INTERNET ACCESS: One common problem is that SMTP e-mail services are often in competition with the web for access from internal networks to the Internet. This means that corporate e-mail to and from the Internet can be interrupted by an excess of casual browser users or PointCast subscribers. To avoid this network planners must anticipate this issue.

7. UNSOLICITED BULK E-MAIL: Most companies are virtually helpless against unsolicited commercial e-mail (colloquially referred to as "SPAM"). Often corporate backbones are flooded by unauthorized SMTP relaying and users are sometimes inundated with SPAM e-mail messages. Spammers use a variety of techniques to probe networks and to discover user e-mail addresses. Companies that fail to aggressively combat spammers could become victims at any point.

8. LARGER AND LARGER MESSAGES: When looking at e-mail statistics network planners often ignore the inevitable increase in message sizes. Also, statistics for internal e-mail are often used to make projections of network traffic for Internet e-mail when, in fact, Internet e-mail messages often have many more and sometimes larger attachments than internal e-mail messages.

9. OVER-SIMPLIFICATION: Capacity plans based on current statistics even allowing 100% increases in 12 to 24 months often turn out to be woefully inadequate. E-mail does not always behave like other systems when it comes to capacity planning because a variety of factors can cause sudden increases and because the peak-to-average ratio for network traffic can vary tremendously from one group of users to another.

10. LACK OF PLANNING: The absence of capacity planning exacerbates all of these issues and is almost always a central factor in catastrophic failures where systems are down, or highly unreliable, for extended periods of time. To make matters worse, when a system begins to crumble, it’s not always clear what the problem is. Tactical fixes are often layered on over time instead of properly planning and undertaking a complete redesign of a company’s SMTP infrastructure.

Number One Warning Sign

Poor performance is the number one warning sign for Internet e-mail. It’s important to remember to test or to attempt to duplicate reported problems such as delivery failures during peak hours.

When a gateway or MTA falls behind in processing messages during peak hours messages queue up faster than they are sent or delivered. When overloaded, the performance of most MTAs and gateways will degrade leading to a rapid reduction in efficiency (an analogy would be to CSMA/CD). The gateway or MTA may catch up slowly during non-peak hours but when users begin complaining that recipients on the net aren’t getting messages in a timely manner it’s time to start looking for the bottlenecks.

Despite the many issues and risk factors for SMTP, Internet e-mail bottlenecks are often to be found in the internal e-mail system. In other words, users may complain about slow Internet e-mail but of course they don’t understand the e-mail system. The actual problem could be in the internal e-mail routing. This should be ruled out before any changes are made or hardware is purchased.

Capacity Planning

Network planners and technical staff often do not know how to predict e-mail traffic on the network or how to justify needed resources to management. Capacity planning for Internet e-mail involves two fundamental things: (1) server sizing, and (2) Network engineering. Here are a couple of pointers:

Typically there are two major utilization peaks for a messaging system associated with the business hours of a given group of users within a given time zone. For a company where most employees work from 9 to 5 the peaks tend to occur at approximately 10:30AM and 1:45PM and last as long as 1 hour. Keep in mind that your actual mileage may vary. Note that the apparent length of peaks could be longer for systems that have insufficient capacity. It’s also important to predict overlapping peaks across time zones if Internet e-mail services are provided centrally for multiple regions.

One maxim of capacity planning is that a server should be sized or a network engineered, to accommodate the peaks rather then the averages. In other words, if messaging statistics tell us that there are 50,000 e-mail messages per day to and from the Internet and, for example, we anticipated that peak traffic would be 200% of average then we have to engineer a system that could handle a peak of approximately 3.5 MPS (messages per second) given an 8 hour day. What that means for servers and networks depends on the e-mail technologies involved, the number of recipients per message, and of course the sizes of the messages themselves. Here is an example model for network traffic projection:

Number of users (theoretical)	2,500	users
Total messages per day sent and received per user	20	messages	(estimated)
Percentage of all messages to and from Internet	5.00%	of messages
Internet messages per day sent and received	2,500	messages
Number of recipient domains per outbound message	1.5	recipients
Additional outbound messages for multiple recipients	625	messages
Internet messages per day sent and received	3,125	messages
Average size of messages to and from Internet	68,619	bytes	67	K
Total message bytes transferred daily	214,434,375	bytes	209	MB
TCP/IP, DNS, SMTP, protocol and transmission error overhead	10.00%	overhead
Total bytes transferred	235,877,813	bytes	230	MB
Length of business day (excluding overlapping time zone hours)	16	hours	57,600	seconds
Network traffic to and from the Internet	1,887,022,500	bits
Number of time zones	5	time zones
Number of overlapping time zone peaks	2
Average network traffic	32,761	bps	32	Kbps
Peak network traffic is n% of average	300.00%	of average
High average network traffic with non-overlapping time zone peaks	98,282	bps	96	Kbps
Peak network traffic for overlapping time zone peaks	137,595	bps	134	Kbps

What’s Next?

This article only scratches the surface for Internet e-mail and capacity planning but it does set the stage for solid analysis and planning by pointing out key considerations and practical tips. Next month we’ll have a look at coexistence and migration of SMTP e-mail services for Lotus cc:Mail and Domino.

Glossary of Terms

CSMA/CD – Carrier Sense Multiple Access/Collision Detection
MAP4 – Internet Mail Access Protocol version 4
LDAP – Lightweight Directory Access Protocol
MIME – Multi-purpose Internet Mail Extensions
MPS – Messages Per Second, a relative measure of e-mail traffic used for server sizing
POP3 – Post Office Protocol version 3
S/MIME – Secure MIME (see MIME)
SMTP – Simple Mail; Transfer Protocol
SSL – Secure Sockets Layer, a standard for the encryption of IP datagrams
X.509 – A standard for digital certificates and encryption used in S/MIME