The fall dinner-party circuit is almost upon us and as I’m sure you know, the hot topic of conversation among the socialite jet-set crowd this season is expected to be database benchmarks. Who know that such a seemingly mundane, esoteric topic could be so interesting?
OK, so I’m exaggerating a bit. Database benchmarking might not make Oprah’s book-of-the-month club. But still, you’ll want too keep up with the latest developments on the pending TPC-E spec to impress all your database geek friends, right?
It’s been a few years since I’ve written much about Transaction Processing Performance Council (TPC) tests. I used to write about the TPC all the time back in the late 1990s and early 2000s, when database vendors made a big show of leapfrogging each other’s TPC-C scores every other week and when Microsoft was demonstrating that it could play with the big boys when it came to scalability. But I’ve never thought that TPC numbers had much relevance for practical, apples-to-apples comparisons of database servers or hardware platforms. Most vendors go to great extremes to avoid posting numbers that make them look bad, so you’ll rarely find TPC comparisons of truly similar systems.
The TPC-C has been the standard test for measuring the performance of “typical” OLTP systems. However, few database environments are “typical,” so many people consider the TPC-C to be a bit too simplistic for measuring or simulating realistic OLTP workloads. This summer, the TPC published a draft specification for TPC-E. Here’s what TPC says about the new specification:
“TPC Benchmark E (TPC-E) is a new On-Line Transaction Processing (OLTP) workload being developed by the TPC. The TPC-E benchmark simulates the OLTP workload of a brokerage firm. The focus of the benchmark is the central database that executes transactions related to the firm’s customer accounts. Although the underlying business model of TPC-E is a brokerage firm, the database schema, data population, transactions, and implementation rules have been designed to be broadly representative of modern OLTP systems.”
You can read the full specification, as well as some supporting documentation and reference material, at http://www.tpc.org/tpce/tpc-e.asp . The full spec weighs in at 271 pages, so I wouldn’t exactly call it light reading. But the site does let you download Egen, a tool designed to facilitate the implementation of TPC-E.
So what does all of this really mean to the average database professional? Not much, in the short term. However, it will be interesting to see whether vendors adopt the new benchmark and publish useful numbers based on TPC-E results. I’d also love to see the TPC invest resources in making Egen a practical tool for regular database shops to do better and more efficient benchmark tests against regular database servers.
