Server virtualization technologies have become so commonplace that they're the de facto standard for server deployment in many organizations. It's becoming more and more common to run into data center environments that operate with the assumption that all new servers will be deployed as virtual guests—unless there's some specific reason not to virtualize. This situation is a significant change from even just a few years ago when the question was reversed, and servers were deployed on physical equipment unless there was a specific reason to virtualize.
So, what about Microsoft Exchange Server? Should you virtualize some or all of an Exchange environment and take advantage of the consolidation, optimization, and flexibility options that virtualization infrastructure provides? The reality is that Exchange environments, particularly those running Exchange Server 2010, can be robustly deployed on virtual servers as long as sufficient resources are allocated to virtual guests and the virtual hosts are scaled correctly. Deploying Exchange improperly in a virtual environment can lead to slowness and other performance problems, and can decrease management confidence in virtualization as a whole, so it's vital to review virtualization design criteria for Exchange in advance.
Microsoft's Virtualization Support Story
The support story for Microsoft products running on virtualization hardware is long and complicated. Until several years ago, Microsoft offered limited support for its flagship server products, such as Microsoft SQL Server, SharePoint, and Exchange, and left open the option that a support problem might need to be duplicated on physical hardware if support technicians couldn't determine the nature of the problem in a virtual environment. Adding to Microsoft's weak support story around the time Exchange Server 2007 was released, Microsoft's own virtualization product at the time, Virtual Server 2005 R2, wasn't a hypervisor-based product and couldn't virtualize 64-bit guests. Therefore, support for virtualization of a 64-bit–only product such as Exchange 2007 was lacking.
Two significant developments changed this story: The first was Microsoft's release of a 64-bit–capable hypervisor, Hyper-V; and the second was the development of a program called the Server Virtualization Validation Program (SVVP), which outlined Microsoft's official support stance on running its products on third-party hypervisor virtualization platforms, such as VMware ESX Server and Citrix XenServer. This program, outlined in the Microsoft article "Support policy for Microsoft software running in non-Microsoft hardware virtualization software" (support.microsoft.com/kb/897615), allowed for support of Microsoft products on third-party virtualization products that were validated by Microsoft and complied with certain criteria, namely the ability for guest sessions to have direct access to hardware resources via a virtualization hypervisor. These two developments opened the floodgates for Microsoft servers running on virtual guests and gave peace of mind to organizations that needed to deploy supported solutions.
Exchange Virtualization Support
Exchange has been a stubborn convert for virtualization over the years. One of the main reasons for this is the fact that Exchange Mailbox servers have traditionally had high performance and disk I/O requirements, which can sometimes be a challenge to replicate on virtual hardware. Although possible to run in a virtual environment, Exchange would consume such a large portion of the resources of the host that the consolidation benefits of virtualization would be minimized. Combine this situation with Microsoft's support stance, and you can see why Exchange server has been one of the applications that has trailed many others in the number of virtualized systems in production.
Exchange Server 2003 was seldom virtualized, except for a few isolated front-end Outlook Web Access (OWA) servers or bridgehead servers. Even in those cases, Microsoft didn't support many of the configurations. Exchange 2007 was the first version to gain broad virtualization support, often at the Client Access and Hub Transport server role tiers but rarely for the Mailbox server role. Exchange 2010, however, is positioned as a great version to virtualize because of both virtualization technology advances and reduction of disk I/O requirements for the Mailbox role that Microsoft has achieved. Disk I/O in Exchange 2010 is significantly less than it is in Exchange 2007, which in itself is less than an equivalent Exchange 2003 server. As a result, many organizations are looking at virtualizing their new Exchange 2010 systems.
There are some limitations to virtualization of Exchange 2010 server roles. For example, Microsoft doesn't support virtualization of the Unified Messaging role. All other roles, however—Client Access, Hub Transport, Edge Transport, and Mailbox—are supported, provided the virtualization software is a member of the aforementioned SVVP.
In addition, Microsoft doesn't support using the Exchange 2010 native mailbox high-availability option, database availability groups (DAGs), on virtual hosts configured for host high availability. This limitation applies to technologies such as VMware's V-Motion, Citrix's XenMotion, Hyper-V Live Migration, and any other solution that automatically fails over the guest sessions from a downed host to another host server. In general, the preferred option would be to use DAG replicas, which have significant advantages, and simply deploy hosts that house the Exchange Mailbox servers without the high-availability options in the host virtualization software.
Virtualization Host Requirements
The key to a stable and high-performance virtualized Exchange environment is using the proper architecture in the virtualization hosts. Out-of-the-box settings and slow disks might work for a test environment, but Exchange has very specific requirements that need to be built into the host system for proper performance to be achieved. Therefore, be sure to follow these minimum requirements when you design the virtualization host infrastructure:
- Allocate sufficient memory for the host OS. If you're using Hyper-V, you'll need to reserve at least 1GB of RAM for use by the Hyper-V host. If you're using a third-party virtualization product, check with the individual provider to determine the minimum amount of memory required.
- Allocate a dedicated NIC for management. Each host should have a dedicated network card for host management, separate from the NIC or NICs used by the virtual guests.
- Allocate a dedicated NIC for failover. If you're using virtual host failover software, such as Hyper-V Live Migration, use a dedicated NIC for the failover. Remember, however, the support limitations for using virtual host failover with Exchange DAGs.
- Use multiple LUNs or storage arrays. Best practice is to allocate a dedicated LUN or storage array for the host OS, another for the guest OS virtual disks, and at least one more for Exchange Mailbox server database storage.
- Use fixed-size or pass-through VHDs. To be a supported Exchange solution, Microsoft requires all Virtual Hard Disks (VHDs) used by Exchange servers to be either fixed-size or pass-through disks that are connected directly to a volume on the host storage. Pass-through disks give you the fastest performance, which is highly recommended for Mailbox servers. Fixed-size disks are faster than dynamically expanding disks, which can suffer performance impacts when they're resizing.
- Use a 2:1 ratio of virtual processors to physical cores. For a virtual host to be supported for Exchange, it can't be overloaded with too many guest sessions. Microsoft caps its support levels on a single virtual host at a 2:1 virtual processor to physical core ratio. In other words, if your host is a 2-processor quad-core system (8 cores total), the maximum number of virtual processors that can be allocated and running at any one time is 16. If each guest session is allocated 4 virtual processors, for example, that caps the number of running guests at 4 on that host.
In addition to these technical specifications for the virtualization host, you'll need to keep in mind some limitations as you set up your virtual environment. Microsoft supports Exchange servers on virtual hosts that are running only virtualization software, with two exceptions made for antivirus and backup software. Overloading the virtual host with other software or other server roles can significantly degrade guest performance. In addition, from a Windows Server licensing perspective, running any roles other than virtualization roles on a Windows server requires one additional license. If the host runs only virtualization roles, however, the host OS isn't counted when determining the number of Windows licenses that are used as part of Microsoft's virtualization licensing program.
Don't combine the Mailbox, Hub Transport, and Client Access server roles on the same virtual guest. An Exchange server with these three roles installed on it would require more virtual processors than can be allocated to it within a single virtual guest, so although it's possible for testing, it isn't supported in production to create a virtual guest with all roles installed. Instead, use at least two virtual guests, one for the Mailbox server and the other with the Hub Transport and Client Access servers. And finally, Microsoft doesn't support reverting an Exchange server to a virtualization software snapshot or using differencing or delta disks.
As a general rule of thumb, it's always a good idea to give as much memory and as many processor cores to your virtual hosts as budget allows. Virtual hosts with multiple multicore processors and large amounts of RAM (64GB, 128GB, or more) are becoming commonplace because of the ability of the virtual host software to take advantage of the additional resources and also because host failover solutions require additional resources on each host to fail over sessions. In reality, there's a sweet spot when it comes to sizing virtualization hosts that balances the cost of the various components against the need to have fewer hosts. Generally, the virtualization overhead required to run virtual servers is only 5 percent, so the resources used by virtualization are more than made up by the advantages of using it.
Software and Licensing Notes
It's highly recommended to use the latest virtualization host software from your particular vendor. The latest version of Hyper-V is included with Windows Server 2008 R2. This version of Hyper-V has significant performance improvements over the original version of Hyper-V and has new features such as Core Parking, Live Migration, I/O improvements for fixed-size VHDs, TCP Offload and Jumbo Frames, and support for Second-Level Address Translation (SLAT)–enabled processors. If you're virtualizing Exchange on Hyper-V, also consider deploying the virtual host on Server Core to minimize its security footprint and memory use.
If you're managing multiple virtual host machines, centralized management software is also recommended. For example, Microsoft offers System Center Virtual Machine Manager (VMM) 2008 R2 for virtualization management. VMM 2008 allows for physical to virtual (P2V) server migration, server template libraries, and management of both Hyper-V and VMware hosts and guests through a single console.
Microsoft provides for cost-effective virtualization licensing options for Windows Server, which lets organizations save significantly on Windows Server licenses when virtualizing servers. The three types of virtualization server licensing are:
- Windows Server Standard Edition, which allows for a single physical OS environment (POSE) or a single virtual OS environment (VOSE) with each Standard Edition license. Note that a virtualization host that's dedicated to virtualization tasks doesn't consume a license even if it's running Windows Server (such as in the case of Hyper-V).
- Windows Server Enterprise Edition, which allows for up to four VOSEs to be run at any one time on the host. Note that only running guests are counted against the license count, so if a guest is shut down, it doesn't count against the total.
- Windows Server Datacenter Edition, which is a per-processor license for the virtual host (a dual quad-core server would require two licenses) that grants you the right to run an unlimited number of virtual servers on the host.
These licensing options apply not only to Microsoft Hyper-V but also to any virtual host that's a member of the SVVP. For organizations with a significant investment in virtualization infrastructure, it's most cost-effective to simply buy an appropriate number of Datacenter Edition licenses to cover all your virtual servers running across your infrastructure.
Virtualization on the Hub and on the Edge
The best virtualization candidates of the Exchange server roles are the Hub Transport and Edge Transport servers. These servers provide for message flow between mailboxes, policy control, and antispam services. They're easily virtualized and in many organizations are the first of the Exchange roles to be virtualized.
Table 1 shows resource guidelines for the Hub Transport and Edge Transport server roles. The rule of thumb is to deploy one Hub Transport server for every five Mailbox servers and allocate 1GB of RAM for each virtual processor when the server is virtualized. These guidelines apply to Edge Transport servers as well. The typical virtual Hub Transport or Edge Transport server is allocated four virtual processors and 4GB of RAM, along with three VHDs—one for the OS, one for the transport database, and one for the transport logs. Best performance can be achieved by placing the transport database and transport logs on separate VHDs that are running on separate spindles on the host OS. If a Hub Transport or Edge Transport server with these specifications needs to handle more mail traffic, you can simply allocate additional servers with the same metrics. The size of the host OS disk should be at least 12GB plus the total amount of memory allocated to the guest, but it's good practice to size this volume larger, typically around 50GB to allow the host OS to grow in size. These recommendations go for any virtualized Exchange server role.
Client Access Server Virtualization
The next likely candidate for virtualization is the Client Access server role, which handles HTTP, HTTPS, IMAP, POP, and MAPI requests directly from clients. This role has become more important in Exchange 2010 because all client connections are proxied through the tier of Client Access servers, making them critical for access to Mailbox servers, and also increasing their resource requirements.
Table 2 shows resource guidelines for Client Access servers. Note that the RAM requirements are double that of the Hub Transport and Edge Transport roles and the number of Client Access servers per Mailbox server is also much higher. Be sure to plan an adequate number of Client Access servers, virtualized or physical, for Exchange 2010. The typical virtual Client Access server role system consists of a virtual guest with four virtual processors and 8GB of RAM allocated to it. It has a single pre-sized 50GB VHD for the guest OS.
Combining the Hub Transport and Client Access Roles
It's common in many organizations to combine the Hub Transport role with the Client Access role to create a combined Hub Transport/Client Access server or set of servers. These servers can then be load balanced using either software or hardware load balancers to provide for high availability of the Client Access service.
Although combining the two roles results in additional load on an individual server session, many of the same processor and memory guidelines that apply to a Client Access server apply to a combined server, as Table 3 shows. The difference lies in the ratio of the number of Hub Transport/Client Access servers to Mailbox servers, which is increased to 1 to 1.
The typical virtual Hub Transport/Client Access server role system consists of a virtual guest with four virtual processors and 8GB of RAM allocated to it. It has a single pre-sized 50GB VHD for the guest OS and two additional pre-sized VHDs for the Hub Transport's database and logs, respectively.
Mailbox Server Virtualization
The Mailbox server is the most sensitive server role to virtualize. This is the server that needs the lion's share of RAM and processor allocation. A physical Mailbox server has the same processor limitations as the other servers do (12 cores maximum) and the same virtual limits (4 virtual processors maximum), but the memory allocation becomes more complex. The equation that's generally used when sizing the Mailbox server is to start with a minimum of 4GB of RAM, then add between 3MB and 24MB of RAM per mailbox, depending on how many messages each mailbox sends and receives daily. Using the information in Table 4, you can determine approximately how much RAM to allocate to a server and how many mailboxes can be housed on each virtual processor allocated to the box. The average mail flow within most organizations is reflected in the table on the row labeled Typical. Your exact mail-flow statistics should be used to determine specifications for your organization.
To illustrate with an example, if you had an organization that averaged 150 75kb messages a day per mailbox and had 2,000 mailboxes, you could deploy a single Mailbox server for all the mailboxes you needed. You could run up to 720 mailboxes per virtual processor, so you would deploy four virtual processors, and you could fit up to 2,880 mailboxes on the server. Using the RAM equation, you would allocate 22GB of RAM to the server by calculating 4GB of RAM plus another 18GB (9MB × 2,000 mailboxes). Of course, you might decide to deploy more servers to provide for DAG fault tolerance or simply to provide a higher level of service, but it's important to understand what Microsoft's limitations are and not to design a system to hold more mailboxes than are listed in the table.
Most organizations don't average more than 100 or 150 75kb messages a day per mailbox; however, there are certain organizations with very high messaging requirements. These organizations should take heed of the higher memory requirements for the Mailbox role and the reduced number of mailboxes that can exist per virtual processor. For example, an organization with extremely high mail loads might consider running just over 1,000 mailboxes per virtual guest and should allocate 26GB of RAM using the earlier calculations.
Keep in mind that these guidelines are simply guidelines. Actual performance will be dictated by the type of disk, hardware architecture, and other factors. Some organizations calculate their hardware requirements, then simply add additional RAM or reduce the number of mailboxes to be safe.
Sample Virtualized Exchange 2010 Architecture
There are many ways of deploying Exchange 2010 in virtualized environments. Some designs are more common than others, however, and reflect common needs across many organizations. For example, high availability is becoming a must for the critical messaging functionality in Exchange. All the new high-availability options in Exchange 2010 are available for virtual environments and can actually be easier to deploy because of the flexibility that virtualization provides.
Figure 1 illustrates a small virtualized Exchange 2010 environment with all components running on a single virtual host. This type of deployment doesn't have any built-in high availability or disaster recovery, but it's the simplest environment to set up and it can still take advantage of virtualization benefits and scalability. Table 5 shows sample server specifications for an environment of this size. These specifications assume 500 mailboxes on the server and an average of 150 messages sent or received per mailbox per day.
Figure 1: A small virtual Exchange 2010 environment on a single virtual host
Figure 2 illustrates a typical virtualization scenario for Exchange 2010 that provides for a very high level of availability, disaster tolerance, and scalability for an environment with 2,000 mailboxes. The entire Exchange environment, including Active Directory Domain Services (AD DS) domain controllers (DCs), is deployed across three virtual hosts—two in the primary data center and one in a secondary data center. DAG replicas of all mailbox databases are replicated to all three servers, providing two backups of each database that can be automatically used in the event of a failure of the Mailbox server holding the active database. Combined Hub Transport/Client Access servers are deployed in each location and are load-balanced in the primary site.
Figure 2: A mid-sized virtual Exchange 2010 environment with high availability
All of these high availability and disaster recovery options are possible without the need for shared storage, a SAN, or host availability solutions. Table 6 lists the sample virtual host and guest architecture guidelines utilized for deployment of the solution illustrated in Figure 2.
Virtualization technologies allow for a high degree of scalability and aren't limited to small- and medium-sized organizations. For example, the architecture that Figure 3 shows allows for tens of thousands of mailboxes, full disaster tolerance, and high availability, all with the high performance expected from Exchange. In this particular model, multiple DAG replicas of Exchange mailbox databases are spread across both virtual guests and a single physical Mailbox server for backup purposes. The server role that can't be virtualized, the Unified Messaging role, is kept on physical servers, and dedicated Hub Transport and Client Access servers are deployed as well. Finally, the management infrastructure for Exchange includes System Center Mobile Device Manager (MDM) and VMM servers integrated into the design.
Figure 3: A large Exchange 2010 organization using virtualization and physical hardware for high availability and management
These three samples illustrate some of the potential design options that are available for a virtual Exchange environment. Every environment is unique, of course, and specifics will vary based on business and technology needs. However, you can use these sample architectures as a starting point for developing a high-performance virtualized Exchange 2010 environment.
Advantage: Virtualization
Server virtualization can provide significant advantages and can let Exchange architects design highly available and disaster-tolerant environments more easily than could be done solely on physical hardware. In addition, virtualized environments have consolidation, optimization, and cost-saving benefits that make them ideal for many organizations. With proper thought into host and guest virtualization architecture, you can deploy a fault-tolerant and high-performance Exchange environment that lets you fully capture the benefits of virtualization for your organization.
