Which Question Was It?

Has an exam question ever completely frustrated you? Have you encountered questions so convoluted or poorly written that you couldn't begin to figure out what they were asking? How about questions for which you were absolutely sure that none of the proposed answers were correct?

Microsoft offers some options for registering complaints about exam questions, but don't hold your breath while you wait for results. You can comment on any question at the end of the exam period, which is probably your best option because the exam is still fresh in your mind and you still have any notes you made while taking the exam. If you want to take your complaint a step further, you can follow the instructions that Microsoft provides at its Training and Certification Web site. However, you must download a special form and fill in details about the exam question within 3 days of taking the exam. Because you can't take any notes from the testing center out into the world with you, you must describe your complaint about an exam question from memory. Unfortunately, the post-exam fried-brain syndrome that I wrote about a a few months ago can interfere with your ability to make a strong case.

The Training and Certification Web site also alludes to the possibility that Microsoft will review your score if you think that a specific question caused you to fail an exam. I suspect that Microsoft has to provide such a challenge mechanism for legal reasons, but I can't imagine anyone ever getting satisfaction from this route. First, you can't know that you actually marked an incorrect answer on the question you're challenging. For all you know, you got the question right and failed because of problems with other questions. Second, you don't get any specific feedback about the re-evaluation. Microsoft says that it will re-evaluate a question and your score, and we just have to trust that the re-evaluation occurs. As the Web site says, "the rationale for the decision will be kept confidential and will not be provided. The evaluation will remain a part of Microsoft's records until the exam is retired." Finally, no appeals process exists.

Coming up with good exam questions is hard work. (You can read about the process on the Training and Certification Web site.) I've been involved in exam development at several levels. I participated in step 5, "Alpha Review and Item Revision," for the SQL Server 6.5 exams, and I've worked with several companies that develop exam-preparation software.

A good exam question must be succinct and unambiguous, and the answer shouldn't be obvious. You shouldn't be able to discern the answer after reading only one sentence or even one paragraph of a question; you should have to depend on combining concepts and skills, as you would on the job. The proposed answers are as hard to create as the questions themselves—at least the wrong answers are (The correct answers should be easy, at least for the person writing the questions!). The wrong answers shouldn't be bogus or obviously wrong. When I participated in the alpha review at Microsoft, we disqualified any wrong answers (called distractors) that referred to nonexistent commands or menu items.

To its credit, Microsoft puts a lot of time and effort into the exam development process. You can clearly see the difference between most of its questions, which are well-thought-out, and the questions you find on some practice tests on the market, especially the free ones. One exam development offer I saw seemed, at first glance, quite lucrative. The company sought experts to develop sample questions. When you did the math, however, you realized that the pay amounted to only $5 per question. Two years ago, people writing questions for more established exam-preparation companies made $100 per question. The wide range of what companies pay to develop their questions reflects the difference in quality among the different exam-preparation vendors. Of course, when you buy exam-preparation materials, you don't know how much profit the company makes or how much the question writer receives. All you know is how much you pay for the materials. But you should be able tell the good from the bad, and you should think twice about paying $9.95 for hundreds of questions. As always, caveat emptor.

Hide comments


  • Allowed HTML tags: <em> <strong> <blockquote> <br> <p>

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.