Battle of the Benchmarks

No matter what new features are added to a database product, the one thing you can always count on to accompany a major release is a new onslaught of benchmarking. Benchmarks are serious business in the database industry—a tremendous amount of publicity, product-name recognition, and proof of concept accompanies new worldclass benchmark scores.

At each new release, vendors such as Microsoft, IBM, and Oracle spend hundreds of thousand of dollars putting together systems and teams to make a new set of assaults on established Transaction Processing Performance Council (TPC) benchmarks.The new release is expected to significantly outperform the old release, and if all goes well, the competition too.

SQL Server 2005 is no exception to this rule. Earlier this year, Microsoft and SQL Server 2005 joined the 1 million transactions per minute (tpmC) club and achieved a mark of 1,082,203 tmpC in the TPC-C nonclustered category. This is the best ever Windows score and the best score ever achieved with Intel's Itanium 2 CPU. The hardware that posted this score was a 64-way HP Integrity Superdome using 1.6 GHz Itanium 2 processors running 64-bit SQL Server 2005 Enterprise Edition on 64-bit Windows Server 2003 Datacenter Edition.Although this score is certainly impressive for SQL Server, it nevertheless falls short of the coveted top nonclustered TPC-C spot held by IBM's DB2 Universal Database (UDB) 8.2 with an out-of-this-world score of 3,210,540 tpmC. IBM's top mark was achieved on a 64-way IBM eServer p5 595 system that used IBM's POWER5 processor and the AIX OS (although it's noteworthy that Microsoft's COM+ was used as the transaction monitor).

SQL Server 2005 also posted a new best (overall, not just for Microsoft and SQL Server) in the TPC-C price/performance category.The new price/performance mark was achieved with 64-bit SQL Server 2005 Standard Edition running on a 1-way Dell 2800 server that used an Intel 2.8 GHz Xeon processor and 64-bit Windows 2003 Standard Edition.This system achieved a score of 38,622 tmpC at a cost of 99¢ per tmpC.

Although some people believe that benchmarks are useless because the test conditions don't match the real world or these naysayers'own specific situation, I think that the TPC-C benchmark tests do successfully demonstrate the high-end scalability of a database system and that the price/ performance numbers give you a good idea of the database's value proposition.The tests are conducted by the vendors themselves, allowing each company to use the full measure of their technical expertise. This eliminates any doubts about whether the system was properly installed and tuned.

The first round of SQL Server 2005 benchmarks has been launched, but I'm sure that more rounds are to come.The upcoming crop of dual-core processors is sure to change the TPC landscape once again. Early results of the dual-core Itanium systems show an increase of about 60 percent in processing power over the current single-core systems. Plus, in the area of price/performance, the dual-core x64 AMD Opteron processors are almost certain to set a number of new TPCC benchmark scores.You can find out more about the TPC benchmark scores at http:// and Microsoft's recent SQL Server benchmark news at http://www. marks.mspx.

Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.