Diving for Benchmarks

Reflecting on the depths that the software industry has sunk to these days is alarming. Leading the way down this slippery slope, Oracle has demonstrated that if your recent software development efforts are lagging behind, a little mudslinging can help turn your luck around. Oracle recently admitted to hiring a private investigation firm to "investigate" (dig through garbage, bribe the janitorial staff) three pro-Microsoft trade organizations: the Association for Competitive Technology, the National Taxpayers Union, and the Independent Institute. Oracle OK'd these dastardly deeds under the ends-justifies-the-means guise of ferreting out some clandestine Microsoft ties. But Oracle's tactics undermine the credibility of all of the investigation's findings. The Department of Justice (DOJ) has become our self-appointed protector from Microsoft, but who's going to protect us from Oracle?

A similar competitive software strategy was in full force during July's Transaction Processing Performance Council (TPC) meeting in Portland, Oregon. After a 5-month faultfinding effort, the TPC disqualified SQL Server's industry-staggering TPC-C scores because the test suite violated clause 1.6.3 of the TPC-C specification. This clause requires primary key updateability in the test suite. Although this clause has officially been part of the TPC-C test requirements since creation of the original specifications in 1992, the TPC hasn't enforced the clause for any earlier clustered TPC-C tests, including Oracle's and Tandem's tests. However, those companies' numbers remain official because organizations can challenge TPC-C results only for a limited time after the initial test. In other words, the older numbers are safely past the TPC's statute of limitations. Since the disqualification of its first scores with SQL Server 2000 and Windows 2000, Microsoft has added support for primary key updates in its distributed partitioned views feature and retested. The latest test score hit a new mark of 262,243 transactions per minute (tpmC), which is 15 percent higher than the original score, and reconfirmed SQL Server 2000 and Win2K's scalability for enterprise-class computing.

In the meantime, IBM delivered a powerful blow in the clustered TPC-C database wars with July's killer mark of 440,879 tpmC. IBM used a DB2 Universal Database (UDB) 7.1 on a 32-node Netfinity 8500 R client/server system, with each node using a 4-way 700MHz Pentium III Xeon processor with 4GB of RAM and 218 18GB hard drives. IBM's recent posting sets the stage for some dramatic TPC-C battles between IBM and Microsoft as the shared-nothing clustered database technology moves out of the pure database arena. Now, each vendor can leapfrog the other by simply adding more nodes to the cluster. Oracle, alas, is in distant third place because its shared-disk clustering technology can't match the incredible scalability that IBM's and Microsoft's shared-nothing implementations are demonstrating.

All is fair in love, war, and benchmarks. And challenging TPC-C results is just part of the game. However, we all benefit when the players concentrate more on database technology and less on corporate intrigue.

Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.