Editor's note: As part of his keynote tomorrow morning on "Next-Generation Messaging" at the Microsoft Exchange Connections conference, Tony Redmond will discuss Microsoft's announcement, made late last week, that it has ended its MAPI Messaging Benchmark (MMB) program. Here's what Tony has to say about the significance of this move to Microsoft Exchange professionals.
On November 1, Microsoft announced to its hardware partners that it had terminated its MMB program. Microsoft first introduced MMB with Exchange Server 4.0 in a response to customer demands for performance data that would help them select an appropriate server configuration for a workload. Over time, MMB evolved through several generations to MMB3 (for Exchange Server 2003) and MMB4 (for Exchange Server 2007). Server vendors diligently executed benchmarks on their equipment using MMB and reported results in terms of how many simulated users a configuration could support.
In the early days of Exchange, it was easy to compare the results attained by single-CPU servers from different vendors to figure out who was the performance leader. As time went by, the complexity of high-end servers increased in terms of CPU, memory, storage, and form factors, and the profile of the work done by users varied dramatically. For example, it's easy to simulate the work done by a user who connects to Exchange with a desktop client and spends all day working through a series of tasks-and the original MMB benchmarks did a good job here. It is much harder to simulate the work done by clients that connect intermittently using mobile devices or even those who use Microsoft Outlook in cached Exchange mode.
Another issue is that while it's interesting to understand how a single server performs, that information doesn't tell you how the server will behave in a production environment where it interacts with many other servers of different types. The different topologies used to deploy Exchange created another difficulty for modeling performance, as did the different applications that run alongside Exchange to deliver the complete messaging ecosystem.
Since 1996, Microsoft has tweaked the MMB user profiles and tasks in an attempt to keep up with new servers, new storage solutions, and new ways of working. But in the end, the shift from benchmarking simple Exchange servers that supported a few hundred users to the kind of systems that we have today became too much effort for too little benefit. You got results for a single-server configuration that you probably wouldn't deploy in production. So, apart from vendors being able to claim that their hardware was fastest, the benchmark results weren't useful. Although Microsoft won't publish new benchmarking results, the company decided that they would continue to develop and support tools used for MMB testing, such as JetStress and Load Generator, to allow hardware vendors to continue to use these tools in their own testing.
The upshot is that you won't be able to compare MMB results for one server against another in the future. However, this isn't a bad thing because it will force server vendors to move their focus from creating hardware that delivers the best MMB results to configurations that are more realistic and can be used as the basis for real-world deployment. Expect to see more case studies and reports that describe fully qualified configurations for Exchange incorporating software (Exchange, antivirus, server management and monitoring, backup, mobile-device management, and so on) plus all aspects of the hardware (CPU, memory, storage, and form factor). Given the need to control power and cooling costs in the data center, you may even see some vendors provide a cost assessment of what it takes to run one type of server configuration against another, much like what we see for refrigerators, cars, and other domestic devices today. But always remember that, just like cars that promise a high gas mileage, achieving the promised performance depends on how you install, deploy, and manage your servers.
