For years, Microsoft Exchange Server designers have worried about the number of I/O operations per second (IOPS) that users generate, especially on large servers. One problem is that the Jet database engine that's the heart of the Exchange Information Store has a 1GB cache that can't be extended on 32-bit systems. Another problem is that Jet organizes data into a series of 4KB pages—and has done so since Exchange Server 4.0 shipped in 1996. At that time, the average corporate message size was around 4KB, so the Store could pack entire messages into one page. Now, the average message size is much larger (more than 100KB in some organizations), and the Store can hold only a small percentage of messages in each page.
Accordingly, the Store processes more I/O requests to read and write messages across multiple pages. We know that messages are unlikely to get smaller as attachments grow larger, users compose message bodies by using graphical editors such as Microsoft Word, and Microsoft starts to push unified-messaging technology so that our mailboxes include voice messages as well. Larger mailboxes and larger databases mean a larger demand for I/O, with the typical design figure for Exchange Server 2003 being 0.5 to 1.0 IOPS per user.
All this is fine when you need to run only a small server. Current server and disk technology can easily and cost-effectively support several hundred Exchange mailboxes. However, when you start to scale up to servers that support more than 1000 mailboxes, sheer storage capacity is no longer the limiting factor nor the most costly item. The crucial factor becomes the storage subsystem's ability to process the I/O demand that users generate. If the storage subsystem can't handle that demand, users see degraded performance because of bottlenecks in the Store.
You can design and deploy storage subsystems that can deal with 4000 concurrent users on an Exchange server, as long as you're willing to spend a lot of money. The expense of providing sufficient storage for large Exchange servers is one of the reasons why Exchange 2003 tops out at around 4000 mailboxes per server (other reasons include the time necessary to perform backup and restore procedures). Storage expense is a real constraint to building larger Exchange servers. Microsoft is more than aware of the I/O problem, if only because the company has gone through a massive consolidation process for its internal Exchange deployment.
