Simple Mail Transfer Protocol (SMTP) is the cornerstone of messaging interoperability across the Internet. Microsoft Exchange Server has always been able to send messages to other SMTP servers by using the Internet Mail Connector (IMC), an optional added-cost component for Exchange Server 4.0, and the Internet Mail Service (IMS), a standard component available to all Exchange Server 5.5 and 5.0 servers. Although anyone can now opt to install the IMS, SMTP connectivity isn't at the heart of earlier Exchange Server versions. The IMS is like any other connector to email systems (e.g., Lotus Notes, cc:Mail, IBM PROFS) and functions as a partner service to the X.400-based Message Transfer Agent (MTA), which makes all the decisions about how to route messages within an enterprise. Routing is a process that assesses the addresses in message headers and decides how best to transport the messages inside and outside a messaging system's boundaries. In Exchange Server 5.5 and earlier, the MTA performs this function. In Exchange 2000 Server (formerly code-named Platinum), a brand-new SMTP-based routing engine takes over.
The architecture that groups different connectors around the MTA goes back to the earliest designs for Exchange Server and reflects the position that X.400 once held as the standard for messaging interoperability. Currently, SMTP holds that position—specifically, the enhanced versions of SMTP that the major messaging systems now implement. SMTP is at the heart of Exchange 2000's message routing and will keep mail flowing during your migration from older Exchange Server systems.
Before Exchange 2000 hits the streets, you need to understand SMTP's history and evolution, its integration with Exchange Server, its role in routing and administration, and its future in the messaging realm. You also need to know how routing groups logically group servers together for the process of sending email. I've based this article's technical information on Exchange 2000 beta 3. The graphical appearance of the screens might change before the software's final release. However, the details about how routing works is unlikely to vary from what I describe.
SMTP's History and Evolution
In 1982, the Internet Engineering Task Force (IETF) defined SMTP in Request for Comments (RFC) 821 and RFC 822. True to SMTP's name, the original protocol was simple and concentrated on the task of sending 7-bit plaintext messages across an IP link between a client and a server. Port 25 is the default port for all SMTP operations.
Since 1982, SMTP has evolved to accommodate the requirements of today's messaging environment. Not surprisingly, given the Internet's pace of change, evolution has been brisk in recent years. Extended SMTP (ESMTP) and Multipurpose Internet Mail Extension (MIME) are the two major advances that have enabled SMTP to deliver highly functional messaging systems.
ESMTP. ESMTP lets vendors add extensions to the basic SMTP service so that they can deal with different classes of messages or other functionality. Exchange Server 5.5 supports ESMTP also because it allows easier pickup of messages queued at ISP mail systems. You need this capability if you connect to the Internet through an ISP and aren't online all the time. The ISP holds messages for your domain until a connection exists, at which point you can instruct Exchange Server to fetch the messages for delivery to local recipients.
Earlier versions of Exchange Server support some ESMTP commands. (For more information about ESMTP, go to the Simpler-Webb Web site at http://www.swinc.com.) For example, to support SMTP over dial-up connections, Exchange Server 5.5 added the extended turn (ETRN) command extension, which also simplifies message dequeuing. (For more information about ETRN, check out http://www.exchangepop3.com/ep3/support.faq/r.faq.general.whynotuseetrn.html.) To support authenticated and encrypted connections between Exchange Server and other SMTP servers, Exchange Server 5.5 Service Pack 2 (SP2) and later versions added the AUTH=LOGIN, STARTTLS, and TLS extensions. To determine the level of ETRN support that an SMTP server offers, you can use Telnet to connect to port 25 and issue an EHLO command. The server responds with a list of keywords for the extensions it supports. Screen 1 shows the support that Exchange Server 5.5 SP2 offers, and Screen 2 shows the list that Exchange 2000 offers. The most important of these new extensions are CHUNKING, PIPELINING, Delivery Status Notification (DSN), and X-LINK2STATE (which I talk about later).
- CHUNKING supports streamed-mode data transfer, which means Exchange 2000 can send and receive messages in binary chunks rather than line by line. Older SMTP servers, including the IMS, process SMTP messages as if they were composed of lines of text instead of the complex structure of body parts (i.e., the pieces that make up a message) we see today. From a performance perspective, the ability to process message data on a binary basis is important—especially as attachments grow larger.
- PIPELINING also aids performance.
It lets the SMTP server issue a set of commands without pausing to wait for an acknowledgment from each command. - DSN lets Exchange 2000 provide standard delivery and nondelivery notifications to users. These notifications replace some of the esoteric and cryptic content that Exchange Server 5.5 sends to users when messages fail to get through via the IMS.
MIME. Sending plaintext messages can be boring, especially in today's graphical age. Early encoding schemes such as UUencode let correspondents exchange attachments and formatted text, but these schemes typically weren't well integrated. Also, you needed to precisely configure the tags that different attachments used before you could confidently send an attachment that the recipient could understand.
MIME solves this configuration problem by establishing rules for the labeling and transmission of different data types within messages. Basically, MIME tells mail systems how to process body parts so that recipients see exactly what the sender intended. You're probably accustomed to using MIME to send items such as Microsoft Word documents and Microsoft PowerPoint presentations between companies. MIME is also the basis for the transmission of streaming data such as audio and video messages, which tend to be much larger than documents or presentations. A famous example is the 25GB Phantom Menace trailer, which caused many email systems to gasp for breath as users forwarded the trailer to friends.
Connector limitations can restrict the size of outgoing messages and stop users from sending such large attachments to other companies; however, circulating large attachments can stop or retard the flow of internal mail. Administrators justifiably worry about the impact that large messages have on systems and discourage people from sending large voice or video attachments. However, large attachments are the way of the future. You can only expect to see an increasing number of large attachments floating around your network.
SMTP and Exchange 2000
Microsoft built Exchange 2000 on the premise that SMTP is the basic protocol for all server-to-server communications—a radical departure from previous implementations, which use remote procedure calls (RPCs) for most server-to-server communications. The move away from RPCs is welcome. When they work, RPCs are great, but they require a lot of bandwidth to work effectively all the time. When RPCs don't work, large message queues can accumulate very quickly. Moving away from RPCs is part of Exchange 2000's general tendency to replace proprietary Microsoft protocols with Internet protocols. Exchange 2000 uses RPCs only when connecting to a downstream server in a mixed-mode organization. (A mixed-mode organization supports both Exchange 2000 and earlier servers.) This use is logical, because RPC is the only protocol that provides guaranteed availability across every version of Exchange Server.
In earlier versions of Exchange Server, you needed to separately install and configure SMTP support. By contrast, SMTP automatically installs on all Exchange 2000 servers. Indeed, every Windows 2000 (Win2K) server installs a basic SMTP service that lets applications send data between servers. Win2K uses this capability to replicate Active Directory (AD) information between Win2K sites. After you install Exchange 2000, the basic SMTP service extends with new capabilities, including support for mailboxes and the advanced queuing mechanism. Technically, Exchange 2000 upgrades the SMTP service to support Collaboration Data Objects (CDO) 3.0 instead of 2.0. Among the other changes, the advanced queuing mechanism allows the implementation of mailbox and connector restrictions. Installing the first Exchange 2000 server in a forest updates the AD schema to support the additional attributes that these restrictions require. In addition to the mailbox quotas that Exchange Server 5.5 supports, you can now stipulate that a connector is available only to specific users or barred to others, or that messages of a certain size can travel across a connector only at specific times.
The fact that Exchange 2000 extends Win2K's basic SMTP service illustrates the tight integration between the OS and the messaging server. Whereas Exchange Server 5.5 and Windows NT have a close relationship, Exchange 2000 and Win2K connect at many different points at a lower level. This weave of interconnections complicates systems administration and places a premium on attention to detail. To manage an Exchange 2000 system successfully, you must be aware of all of the dependencies and linkages. Screen 3 shows that the SMTP service depends on Microsoft Internet Information Server (IIS), the Exchange Server Information Store (IS), and Exchange System Attendant services. Problems that occur with IIS or the IS might affect the SMTP service. Likewise, if the SMTP service experiences difficulties, you probably won't be able to send messages between your server and other Exchange servers, or to the outside world. An SMTP malfunction will also affect other applications. For example, because replication between Win2K sites can occur through SMTP messages, AD intersite replication might stop working if you misconfigure SMTP on a domain controller.
Virtual Servers
Through IIS 5.0, Win2K implements virtual server support for a selection of Internet protocols, including HTTP, SMTP, POP3, and Internet Message Access Protocol 4 (IMAP4). Think of a virtual server as a combination of a protocol, port number, and configuration data. Multiple virtual servers can run on one physical computer, so you can now run multiple SMTP virtual servers on one Exchange 2000 server. The default virtual server will connect to port 25, but you could equally assign other virtual servers to use other ports and then use those virtual servers to handle email going to different domains. This important piece of functionality lets Exchange 2000 support many different email domains on one server, which is something that ISPs need to do all the time. Additionally, Exchange 2000 supports multiple SMTP connectors on a virtual server. Think of IMS as an SMTP connector. Today, you can run only one SMTP connector on a physical computer, but with Exchange 2000 you can configure multiple connectors, each for a specific purpose. For example, you might configure one connector to route messages to internal systems and another to send all external mail and direct that external mail through a SmartHost system, where a virus checker checks messages before they go to the Internet.
Messaging Administration
Exchange Server 5.5 groups servers into sites—one or more servers connected in a common management unit that uses LAN-quality bandwidth. You connect sites to form an organization, a management unit that shares a common directory and configuration. The organization concept remains in Exchange 2000. Only one Exchange 2000 organization can exist in a Win2K forest because servers now share configuration data through AD's configuration naming context.
Win2K uses the term site to describe a collection of servers that share a common network—in this case, one or more IP subnets define the network. Win2K sites are far more tied to the underlying network than Exchange Server 5.5 sites are. I doubt we'll see a Win2K site that spans the world, but I know of several successfully implemented Exchange Server worldwide sites. (For more information, see Ian Tweedie, "Global Site," July 1999.) Win2K sites don't have the same profound administrative effect that sites have in Exchange Server 5.5, in which a site marks the boundary for administrative, security, and replication operations.
Administrative groups and routing groups replace Exchange Server 5.5 sites. An administrative group sets the boundary for management operations. The administrator manages all the servers in an administrative group according to common policies, which might remind you of the way servers in an Exchange Server 5.5 site share settings (e.g., default site addressing). Routing groups define how messages flow between an organization's servers. Each server in a routing group can make a point-to-point connection to another server on a demand basis in exactly the same way that servers connect in an Exchange Server 5.5 site. In mixed-mode organizations, one administrative group and one routing group represent each Exchange Server site. In mixed mode, Exchange 2000 pretends that it uses the site model. Because older servers don't understand the division between administration and routing, Exchange 2000 must mimic the site structure. Later, when all servers are running Exchange 2000, you can switch the organization into native mode and split the groups to form a different structure.
Sites work extremely well in small organizations in which administrators typically monitor management and routing. However, large enterprises often have teams to take care of server management and the email backbone. The good news is that you can manage Exchange 2000 in the same way that you manage Exchange Server 5.5, if those procedures make sense for your company. But Exchange 2000 gives you greater management granularity by letting you allocate responsibilities to different teams. Greater granularity in systems administration is a theme throughout Win2K. Microsoft designed the Microsoft Management Console (MMC) framework so that you can build customized consoles by assembling snap-ins that meet specific needs.
Many large companies operate multiple levels of support from their Help desk. The Help desk, in turn, takes the support calls to backup support, which might take complex problems directly to Microsoft support. Exchange Server 5.5 provides one tool that everyone can use—the Microsoft Exchange Administrator program. Certainly, a single all-in-one tool is convenient. However, why should someone who simply changes users' directory information have access to a tool that administrators who configure complex components such as X.400 or SMTP connectors use?
Screen 4 shows the Exchange 2000 System Manager, an MMC console comprising several snap-ins that is the closest thing you'll find to Exchange Administrator. The fact that you can allocate the snap-ins to different groups makes granularity possible. The screen shows the relationship between administrative groups and routing groups. An administrative group typically has a broader range and scope than a routing group. In this example, three administrative groups manage a worldwide deployment, and each administrative group has one or more routing groups. The QEMEA (European) Administrative Group spans four routing groups (i.e., France, Germany, Ireland, and Valbonne).
Routing Groups
All the servers inside a routing group can communicate with one another automatically. The same situation exists in an Exchange Server 5.5 site, but the servers communicate differently. Remember that Exchange Server 5.5 servers use RPCs, whereas Exchange 2000 servers use SMTP and connect to the standard port 25. Another basic difference between Exchange Server 5.5 and Exchange 2000 is that directory replication generates much of the intrasite traffic between Exchange Server 5.5 servers, whereas the vast majority of email that users send between Exchange 2000 servers is interpersonal communication. Exchange 2000 sends a small amount of data to update servers about the state of the network, but this data represents far less overhead than the overhead of a site's directory replication. Directory replication is now AD's responsibility.
Connecting Routing Groups
You can use the Routing Group Connector (RGC), the SMTP connector, or the X.400 connector to connect routing groups. In some respects, the RGC is the equivalent of Exchange Server 5.5's RPC-based Site Connector. The RGC is easy to set up and configure and is typically the connector of choice. Unlike the Site Connector, the RGC offers you a finer degree of control over the servers that participate in routing. The Site Connector lets you select all servers in a site or a nominated bridgehead server to handle traffic for the site. The RGC lets you nominate as many servers as you want to act as a bridgehead. The RGC also lets you restrict the connector to particular users, define the priority of traffic across the connector (i.e., high, medium, or low), and determine the connector's availability. You can also establish a schedule to control when large messages go across the connector, so you can prevent large attachments from interfering with typical business traffic.
The RGC handles only SMTP traffic and runs under the context of an SMTP virtual server. The RGC is a unidirectional connector, and you must configure both sides of the connection before messages can flow both ways. However, if you have permission to perform administrative operations in the target routing group, you can set up both sides of the connection in one step.
Screen 5 shows the properties of an RGC linking the Ireland routing group with the Valbonne routing group. The Do not allow public folder referrals check box is important. By default, the check box is cleared; therefore, users can connect to public folder replicas on a server in another routing group across this connector. In Exchange 2000, referrals replace Exchange Server 5.5's concept of public folder affinity, and routing groups now achieve the notion of control that sites and locations (or subsites) previously accomplished. When users attempt to access a public folder, Exchange 2000 first attempts to refer the users to a replica of the public folder that resides on the same server as their mailbox. If a replica is unavailable, Exchange 2000 then refers the users to another server within the users' routing group. If a replica is still unavailable, Exchange 2000 attempts to refer the users to a replica of the public folder that resides in a routing group across the lowest-cost connector. Each connector has a cost (i.e., from 1 to 100) that rates the expense (in network terms) of connecting to a target bridgehead. Lower values indicate a preferred route. In the case of the RGC that Screen 5 shows, the cost of the connector is 1, the lowest possible value. This connector is a preferred route to a public folder replica if the replica is available in the Valbonne routing group.
The RGC uses SMTP, so why do we need a separate SMTP connector? The answer is simple—Exchange 2000 is still a small presence in the SMTP world. People want to communicate with other domains that use SMTP, and they'd like to do so in as functional a manner as possible.
Like the RGC, the SMTP connector runs under the context of an SMTP virtual server. The RGC uses the Exchange 2000 configuration data in AD to route messages, whereas the SMTP connector uses standard SMTP routing techniques such as the mail exchanger (MX) records in DNS. Therefore, whenever you want to route messages to a SmartHost system, use MX records as the basis for routing, or simply send messages outside Exchange Server (including to the IMS running on older Exchange servers), you need to use an SMTP connector.
Screen 6 shows an SMTP connector in action. This connector sends all messages to smarthost.compaq.com. Smart hosts are mail servers that often act as central collection and dispatch points for an enterprise's SMTP mail. Bringing messages together to a central point before they leave the company lets you check the messages for viruses and profanity. Smart hosts also let you verify that users are complying with company standards (e.g., attaching disclaimer text on messages, refraining from sending classified information to external correspondents). MIMEsweeper (http://www.mimesweeper.com) is a popular example of software that runs on smart hosts.
Setting up the connector to route all mail to a smart host is one step toward control. You can achieve finer control through the SMTP connector's Address Space property sheet, which is where you define the SMTP domains that Exchange 2000 can reach through the connector. The default setting is an asterisk (*), which means that Exchange 2000 can reach all domains. However, in some cases, you might want Exchange 2000 to handle a specific domain's messages in a certain manner. Screen 7 shows the address space configured for a specific domain (i.e., compaq.com). With this configuration, the connector can only handle messages addressed to a recipient in compaq.com; all other messages must find another route. At the bottom of the dialog box in Screen 7, you can see how Exchange 2000 controls the connector scope. Exchange Server 5.5 introduced the Connector scope feature, which restricts the availability of a connector to other servers. In Exchange Server 5.5, you can restrict a connector to servers at a location, site, or the organization. In Exchange 2000, you can restrict either to the routing group or the organization. In Screen 7, the connector is available only to the servers in the local routing group.
Link State Routing
A connector is simply a link between a routing group and another set of systems. Multiple routing groups and connectors can operate within a company. Therefore, to foster effective decision-making about message routing, you need a mechanism that informs all the servers about these other servers and connectors.
To maintain a list of all possible routes within an organization, Exchange Server 5.5 uses the Gateway Address Routing Table (GWART). A server called the Relative Identifier (RID) master (typically the first server you install in a site) builds the GWART nightly, using information replicated around the organization. The other servers in the site automatically share the GWART. Thus, every server in a site has a common understanding of the available connectors and the best way to route messages.
Exchange Server has used this mechanism since 1996. The GWART is fairly successful, but its nondynamic nature is a major drawback. The GWART works best when the network is stable and servers and connectors are available in a predictable manner. If inoperative connectors introduce a level of unpredictability into the routing equation, the GWART loses some of its effectiveness and the potential for misrouting grows. Misrouting doesn't result in lost messages. However, misrouting can send messages down paths that might lead to a dead end. Such an occurrence requires you to reroute the messages so that they can reach their final destination. Rerouting is expensive and slows message delivery, so you need to avoid it.
We can't all have a wonderfully stable network environment, so we need to move toward a more dynamic way of updating servers about the current state of the network and the routes that are available to send messages. Exchange 2000 uses Link State Routing. LSR lets servers update one another about changes that occur to network links or connectors. Information about those changes broadcasts quickly to other routing groups so that their picture of the network is current.
Imagine that a network error causes a connection to a routing group to fail. After 60 seconds, the bridgehead server makes a second attempt to contact the bridgehead server in the remote routing group to determine whether the fault is a temporary glitch. If the connection fails, the bridgehead server tries twice more before concluding that the problem is persistent. The server marks the link as down and establishes a connection (using port 3044) to the Routing Group Master. Each routing group nominates a Routing Group Master to collect updates from the servers in the routing group and also to collect updates from other routing groups. With that information, the Routing Group Master builds a Link State Table (LST), which represents the current state of the routing network. The Routing Group Master is typically the first server you install into a routing group, but you can easily use the Exchange System Manager console to transfer this role, as Screen 8 shows.
After the Routing Group Master receives the update about the inoperative link, it adjusts its LST and broadcasts details about the new LST to all the routing group's servers. Once again, communication takes place through port 3044 using a special protocol called LSA, which Microsoft developed for this purpose. Exchange 2000 uses an ESMTP extension called X-LINK2STATE to send link-state updates between routing groups across the RGC and SMTP connectors. Exchange 2000 sends updates between bridgehead servers, which then pass the updates to the Routing Group Master in the local routing group. Exchange 2000 can also send updates in the form of application-specific data across X.400 connectors. The passed data is highly compressed, so Exchange 2000 needs only a matter of seconds to pass the information and create a new LST for the routing group. The addition of new servers or the creation of new connectors also initiates link-state updates.
Link-state updates ripple quickly through an organization. The aim is to reach every server in an organization within a matter of minutes—rather than hours or days, as can happen today. Updates travel as quickly as messages move between servers. Because Exchange 2000 needs to transfer so little data, link-state updates typically move across extended links before administrators realize that a problem exists.
The Future of Messaging
I've only scratched the surface of Exchange 2000's new technology. In addition to the split between administration and routing, the messaging system offers a host of settings that you can apply to the SMTP service and its virtual servers to make messages flow. Also, to link the servers in an Exchange Server organization together, we have a set of new connectors. If that weren't enough, the topic of link-state routing is substantial enough to warrant a separate discussion in its own right.
Above all, the fact that Exchange 2000 is very different from its predecessors is clear. Given that Exchange 2000 can run only on Win2K and that you must have a solid infrastructure in place before you deploy your first Exchange 2000 server, your knowledge of how SMTP works with Exchange 2000 might take time. However, you need to understand now that SMTP is just one of the close dependencies that exist between Win2K and Exchange 2000. SMTP represents the future of message routing in Exchange Server.
Corrections to this Article:- When Exchange 2000 and SMTP identifies what the acronym RID stands for, it cites the Active Directory (AD) term rather than the Exchange term. In the context of the article, RID stands for Routing Information Daemon, not Relative Identifier.
