I still vividly recall my first few weeks working for Sybase as a technical support engineer. After only three short weeks of training, I was thrown into the lions’ den, meaning I had to start taking phone calls from customers. I remember how excited I was the first time I encountered a verifiable, reproducible bug and got to fill out an official bug report. I also have a very clear memory of my husband’s response when I told him about it that evening: "What?! You mean they are selling a product with bugs in it?"
Working for Sybase was my first corporate job after finishing my degree and teaching introductory programming at a university for a few years, so I had never been involved with the inner workings of a commercial product. But I suppose my education provided me with enough appreciation for the degree of complexity inherent in any product so that I had no problem dealing with the fact that there were bugs. However, it was difficult for me to convey all my understanding to my husband.
Since those early days, I’ve learned even more about the complexity of software and the fact that not only is testing an entire department in many companies, but also that it’s an art and science completely separate from software development. Although I’ve never worked as a software developer, I know how difficult it was to thoroughly test all the software projects my students wrote in the classes I taught, and those projects were many orders of magnitude less complex than a commercial product.
So is bug-free software just a dream? If we accept that bugs are always going to be part of a product, does that mean the doors are wide open to whatever bugs might be left in when the product is released? Might there be a way to classify the "types" of bugs or the number of bugs that would be acceptable, or should we just trust that vendors will do the best they can to eliminate bugs? I certainly don’t have any answers, but I have been coming up with a lot of questions.
As I was thinking about this topic and planning to write this commentary, I came across a post at www.codinghorror.com/blog/archives/001313.html, indicating that other people have the same questions. The author is asserting that it’s fine to release buggy software and let users test it in their real-world environments. Although there are definitely advantages to this approach, the disadvantages should also be fairly obvious, and the comments seem to indicate that other people think so, too.
If we allow "some" bugs of "some" types, should there be a different standard for different types of software? Should tools to help keep your appointments organized be held to the same standard as OS software or your database software? If there are too many bugs in my organizer software, I can relatively easily switch to a competing product, but moving to a new OS isn’t a task to be taken lightly. But with so much more complexity in your OS, there are many more places that bugs can occur. So how much testing is good enough, and how many, and what type, of bugs are acceptable?
One additional issue to consider with regards to buggy software is how communicative the software company is about acknowledging the bugs. You might also consider what kinds of policies the company has for providing hotfixes when bugs are encountered that severely affect your own business running on the software. If bugs are to be accepted as a fact of life in the software world, there should be well publicized procedures that the vendor has for dealing with the bugs that are reported.
Depending on your support contract with Microsoft (if you have one), hotfixes are certainly a possibility to help you deal with critical bugs. As far as acknowledging existing bugs, I believe Microsoft is getting much better about that. The company’s Connect site lets you search for previously reported bugs and report bugs you discover. Microsoft engineers will provide feedback as the bug is processed or report on why the bug can’t be fixed. The Connect site itself isn’t perfect, and neither are the status reports from the engineers, but the ability to communicate with the software engineers is a huge step forward in enabling customers to feel that they’re being listened to. Even without perfect software, we can start to feel like the software will meet our internal needs and the needs of our business.
I wish you a perfectly good enough holiday season and an acceptable New Year.
