NT scalability is a critical topic for Windows NT Magazine, and testing scalability is a regular focus for our Lab. Thus, you would think the Lab Guys were thrilled by Microsoft's Scalability Day last May, right? Excitement was high when the Lab Guys flew into Manhattan, but expectations were modest--after all, Scalability Day had all the earmarks of a well-oiled Microsoft press event. As things turned out, that's exactly what Scalability Day was--a series of slick, well-rehearsed speeches and demonstrations on NT scalability. To be honest, I left New York with more questions than answers. This month, I'd like to brush the dust off Scalability Day to reflect on what attendees saw and, equally important, what attendees didn't see.
The Event
Let's start with the basics. Scalability Day was a two-part event. In part
one, Microsoft herded an audience of about 500 press members and industry
analysts into an auditorium and exposed us to a series of speakers--Microsoft's
Bill Gates (chairman and CEO), Paul Maritz (group vice president, platforms and
applications), Deborah Willingham (vice president, enterprise customer unit),
and SAP's Gunter Tolkmit (vice president, corporate marketing). Each Microsoft
speaker called in additional Microsoft staff to conduct the live demonstrations.
The demonstrations were glitzy, MTV-speed presentations designed for the
attention-challenged (I'll get to the details of the presentations later).
The second part of Scalability Day was a Microsoft Partner Pavilion where you could press the flesh with some of the Solution Partners that Microsoft deemed worthy. This group was eclectic--vendors ranged from high-end scalability hardware manufacturers to financial application software developers. I cannot begin to guess how Microsoft came up with the invitation list, nor can I guess why some vendors weren't invited. Notably absent? IBM comes to mind. (For Mark Smith's view about noticeable absentees, see his July editorial, "Scalability Politics.") The pavilion tour was a decidedly low-key event; much to my disappointment, no scalability demonstrations took place on the show floor. For example, seeing Tandem's 16 * 4 cluster demonstration would have been pretty cool, but instead, attendees were supposed to be content looking at a picture of the 16 * 4 configuration. Sometimes a picture is not worth a thousand words.
Substance or Substance Abuse?
After sifting through the morning speeches and wandering through the
pavilion in the afternoon, I realized that the meat of Scalability Day was the
series of live demonstrations that peppered the speeches. The six demonstrations
were
- A SQL Server system containing 1TB of data: This single server was a quad-processor (Alpha 450MHz) Digital Alpha 4100 system with an attached Digital StorageWorks storage array containing 1.3TB of disk space. The server was running NT Server 4.0, the Sphinx release of SQL Server, Internet Information Server (IIS) 3.0, and Transaction Server 1.0.
- A set of SQL Server systems servicing 1 billion transactions per day: This transaction load was spread over 20 quad-processor (Pentium Pro 200MHz) Compaq ProLiant 2500 systems interconnected via Microsoft Distributed Transaction Coordinator. All the systems were running NT Server 4.0 and SQL Server 6.5.
- An IIS system handling 100 million hits per day: The platform was a dual-processor (Pentium Pro 200MHz) HP NetServer LX system running NT Server 4.0 and IIS 3.0.
- An Exchange Server system servicing 1.8 million email (Post Office Protocol 3--POP3) messages per day: This demonstration had a beta version of Exchange running NT Server 4.0 on a quad-processor (Alpha 466MHz) Digital Alpha 4100 system.
- A planned and an unplanned failover of SQL Server and an SAP application using Wolfpack clustering software. The platform for this demonstration was a two-server cluster from Tandem Computers running the Enterprise Edition of NT Server, which includes support for Wolfpack clustering.
- A realtime comparison of 64-bit NT Server vs. 32-bit NT Server: Microsoft used two identically configured systems for this demonstration. Both systems were eight-processor (Alpha 440MHz) Digital AlphaServer 8400 models running beta versions of NT Server 5.0 and the Sphinx release of SQL Server. One system had 64-bit memory access enabled, and the other system had it disabled.
Were all these demonstrations impressive? You bet! They were flashy, with lots of moving images and buzzing counters. Were they substantive? Well, that's an entirely different matter. Let's look at these demonstrations a little closer.
TB or Not TB?
At the terabyte demonstration, a database on the Sphinx release of SQL
Server contained slightly more than 1TB of data. The bulk of the data was images
of the Earth taken by satellite cameras. The database contained location
information so that you could find an image from geographical input. The lookup
application was Web-based. You could enter the necessary parameters and receive
the corresponding image. The amount of data was impressive, as was the fast
lookup speed, but what did this test really prove?
If you think about the database this test used, the image files ate up most of the terabyte of disk space and were not processed during any of the front-end lookups. Geographical information was associated with the image files, but the amount of geographical information wasn't close to a terabyte. In the final analysis, Microsoft had a fairly small set of indexes pointing to a series of large image files. Scanning those indexes quickly and retrieving a file based on the results is hardly equivalent to reading or writing a terabyte of information.
Was this test completely invalid? No, it wasn't. I was impressed that you can use a SQL Server database to store a terabyte of data. But the demonstration application--the online lookup of images--is not a hard-core, real-world database application. A more impressive test would be to fill the SQL Server database with 1TB of searchable rows and columns. Of course, that kind of test would execute too slowly for the fast pace of the Scalability Day--a task that took minutes to process would have seemed snail-like compared to the pace of the show.
Take It to the Bank
The second demonstration was a simulated banking environment capable of
handling 1 billion ATM transactions per day. As a former application and
networking consultant, I know that ATM transactions are short and sweet. In many
ways, ATM transactions represent the best possible transaction load because not
only are they short, but they involve very few message exchanges between the ATM
device and the server. If I were going to set up a simulated transaction
environment to show a high transaction rate, I'd be inclined to use ATMs, too.
The configuration of the server side of the demonstration, however, was very interesting. Instead of setting up one database to handle all the ATMs, Microsoft grouped the ATMs into 20 divisions (bank branches) and assigned a separate SQL Server to each division. Multiple SQL Servers were interlinked via Microsoft's Distributed Transaction Coordinator. All things considered, this design is pretty good and would stand up in numerous real-world environments; I give Microsoft high marks for the server side of this demonstration.
The demonstration showed that under ideal conditions, you can generate 1 billion transactions today using NT technology. The problem is that nobody lives in these conditions. In reality, the SQL Servers must process standard banking transactions for name and address changes, new accounts, account closings, wire transfers, and other full-size transactions that sap the potential to deliver 1 billion ATM transactions per day. If all you want to do is create an ATM network, you can achieve the advertised rate of 1 billion transactions per day; if you want to run a real banking environment, you will undoubtedly find your performance to be less than the advertised rate.
Real World or Ideal World?
Although the demonstrations were impressive, they were designed to run in
the ideal world, not the real world. This trend continued through three other
demonstrations. The IIS demonstration of 100 million hits per day used static
Web pages, not dynamic pages. The Exchange server mail demonstration used POP3,
not the standard Microsoft mail connection. The Wolfpack failover demonstration
was well orchestrated and hardly represents the dramatic and unpredictable
failures that can occur in our data centers. In short, the demonstrations made
for a good show, but not for good research data.
Don't get me wrong--I'm not saying that the demonstrations were meaningless. Taken in the proper context, they show how NT can perform in the best of times. The problem is, most of us don't live in an ideal world. We all live in the world where networks clog and break, applications are resource hogs, and our users continue to confound and confuse our best business practices. Trying to map Microsoft's glitzy, MTV-style tests into the dirty, gritty reality of our day-to-day data processing lives is an impossible feat.
However, I will admit that the demonstration of the NT 64-bit Very Large Memory (VLM) implementation impressed me. This demonstration showed two NT Alpha machines running the same application (a database lookup); only one of the machines was running with VLM enabled. The VLM-enabled system ran the application unbelievably faster--I was ready to take out my wallet and invest in an Alpha machine with lots of memory right then and there.
Lasting Memories
Scalability Day reinforced the Lab's plan to test NT scalability using
real-world tools and analysis techniques. Our tests are not glitzy and don't run
at MTV speed. A single test takes us hours or days to conduct. You won't see us
performing these tests on stage or on video because they are, frankly, boring to
watch. But they provide us with interesting insights about NT and BackOffice
scalability. You get to skip watching the tests and instead can read meaningful
results in the magazine.