When Microsoft releases a product, it builds in a comprehensive Help system that documents the product's features. But when the company updates the product or fixes bugs (which users ironically call "undocumented features"), the software's Help system doesn't reflect the changes and becomes outdated. To ensure that customers can learn about such changes, Microsoft documents them through Customer Service and Support (CSS, formerly known as Microsoft Product Support Services—PSS), the organization that produces and publishes Knowledge Base (KB) articles on the www.support.microsoft.com Web site.
The CSS support site is the subject of this month's reader survey, which asked how IT pros use that and other Microsoft sites and what site-related questions they have for Microsoft. Survey participants most frequently asked questions about the effectiveness of the support site's search capability. By far, respondents' biggest concern is finding information about errors and events.
Microsoft customers learn to rely on the support site to find solutions to technical problems. In fact, of the 523 survey respondents, nearly 59 percent say they visit this site at least once a week, 28.5 percent at least once a month, 6 percent at least four times a year, just 5.5 percent rarely, and only 1 percent never. Nearly 91 percent of respondents use the site to find information about a specific error message, 80 percent get downloads and updates, 74 percent look for tips, and 72 percent use the site for troubleshooting. (Respondents to the usage question selected all applicable responses from a list.) We also asked participants to rank the steps they take when they have a problem with a Microsoft product. The results, in order of preference, are: Go to Microsoft Web sites (20.6 percent), search the Internet (19.3 percent), access the product's Help files (18.4 percent), ask a friend (14.5 percent), consult a newsgroup or forum (11.2 percent), and contact Microsoft customer support (7.6 percent).
The survey also asked readers to rate the usefulness of each Microsoft site. TechNet came out on top, with about 45 percent rating it extremely useful. Thirty-four percent of respondents rated the support site extremely useful, but only 11 percent gave microsoft.com the same rating. I found it interesting that although our audience consists primarily of IT pros, MSDN fared better than microsoft.com, with 22 percent considering the developer site extremely useful.
This month's survey revealed some confusion about how Microsoft decides whether to expose a particular piece of content on microsoft.com, TechNet, MSDN, or the KB. For an explanation of what distinguishes these sites, see the Interact! sidebar "What Content Is Where?" at http://www.windowsitpro.com, InstantDoc ID 45883.
In Search of the Perfect Search
I discussed these findings and participants' questions with Microsoft's Kurt Samuelson, general manager of Global Service Automation, Customer Service and Support. Kurt was especially interested in the survey data because the support site was recently redesigned based on extensive customer research.
Kurt told me, "Your survey helped validate that we made good decisions with our changes. People wanted better search, and they wanted their information organized—for example, all the KB articles now include a table of contents with links that direct you into the content."
What else has changed? "The old site was task-based—for example, you can search or submit an incident. Customer interviews about problem-solving habits showed that approach was counterintuitive. A lot of decisions drove from there. The new site is now product-based. So if you pick XP from the left-hand navigation bar, for instance, then you're in that context. Everything you do is related to that product: If you search, the product context is selected for you automatically. If you want to navigate to other content, it's all relevant to that product space. That was a fundamental design change that drove through that experience."
Search is definitely an issue that users have with Microsoft sites, and many readers said that they prefer Google. An interesting question was "Why was the KB search engine changed?"
Kurt explained, "One challenge is that we have a broad base of users: IT professionals represent about 20 percent of our users and the general consumer represents about 62 percent, but the IT professional is our biggest repeat user. They use search as a primary means of navigating through information. Often they've made a big investment in understanding keywords and things that make search work well. We've learned something we didn't see in our usability studies that we're seeing now: In practice, consumers aren't ever going to the advanced search. They're just going to the top one."
What's the solution? Kurt said, "I want to satisfy both groups, but they have different habits and skill levels. So we try to find the balance."
What changes benefit IT pros? "One improvement is that we started using a dynamic feedback mechanism to improve relevance. With our new approach, when somebody prints a KB article or emails it to a friend, we note that and it affects the search relevance next time. That approach is improving search results."
Kurt continued, "This spring we're taking that idea of references further. People often refer to KB articles in newsgroups, so we've figured out how to mine references to KB articles in the newsgroups, and that information will affect search relevance."
One reader noted, "The hardest task is asking the right question when there's no specific error message. What are you doing to improve in this area?" Kurt replied, "A big part of relevance is understanding the language that people use to describe their problem, so getting that language from the newsgroups will help us get better. We also look at the statistics on our site and feed that data back into search. Another part is how we design a search interface."
What's new in the interface? "The interface today lets you do a weighted or a filtered search." Kurt explained. "In the past, when you selected a product for a KB search, you'd get content on that product only. Now the default behavior is a weighted search, meaning that if you choose Windows XP, XP articles will bubble up to the top of the results list but related topics aren't excluded. Or you can choose a filtered search in the Advanced Search options, which excludes all the other stuff. That option helps balance between the more expert searcher and the novice searcher."
When is a weighted search important? "Suppose I can't figure out how to print my Word document. Is that a problem with the XP print driver? Is it a problem with Word? Is it a problem with something else? The point is to provide as much specificity as possible without excluding a lot of stuff that might be relevant. It's a fine line between providing specificity to narrow down a search scope and locking out where the answer might actually lie."
One reader noted that search has improved but wants more: "It still can take several attempts to find required information." Kurt explained that the new site has been restructured. "We implemented a solution taxonomy. In the past, you couldn't navigate through the XP or Exchange or Office content. We now have a category hierarchy of topics—for instance, Administration, and under Administration, Setup, and that can go down 3 or 4 levels deep. Then we mapped common keywords into that taxonomy. So if you know your general problem area but you don't know what the issue is, you can click on a series of links to say, 'OK I know my problem is with Exchange.' Then you can see how-to, troubleshooting, and download articles as a way to narrow it down. So if you think you're in the right place but the article isn't the right one, you look on the side of the article at the list of related articles by keyword and topic. You can navigate to those without going back to search."
Error! Error! Error!
A typically frustrated reader asked, "Do you think you could provide support for all error messages that come from your products?" Kurt wasn't surprised by that frustration: "We also see that in focus groups and usability studies. Consistently, customers want to figure out event IDs and error messages. We know it's a gap we need to address specifically. Some ideas are on the table, but I don't have a solution right now except to say we're working on it."
Responding to the survey finding that many respondents go to third-party sites for errors and events (e.g., www.eventid.net). Kurt said that Microsoft wants to cooperate with such sites. "I don't have a specific set of objectives around this yet, but it consistently comes up and we have to solve it in the next year."
The Rest of the Story
What's ahead for the support site? "The next step is to figure out how customers can personalize the support experience. Formerly, we said 'Go try to be successful with this big pile of content called the KB.' Today, we've provided more structure and say, 'You're within your product area now: Here are some other tools to help.' In the near future, we want to provide an experience customized to what you're doing in your work."
This month's survey produced a lot of data. To find Kurt's answers to readers' questions about Most Valuable Professional (MVP) content on the support site and whether Microsoft validates such content, see the Interact! sidebar "Product Support from the Trenches," InstantDoc ID 45884.
Let me know what you think. What products or technologies do you want to ask Microsoft about?
