This week, Microsoft will ship its third Community Technical Preview (CTP) build of Windows Vista to beta testers, MSDN subscribers, and TechNet members. I recently spent the day with various members of the Windows Vista team at Microsoft's Redmond campus, where I received various demonstrations of CTP candidate builds and discussed the future of the Windows Vista beta program. In this showcase, I'll put the December CTP release in perspective, and provide some background about how the CTP program is proceeding, and how it, and the product it represents, will evolve as Microsoft moves towards a late 2006 final release.
Before proceeding, it may be useful for you to examine some of my previous articles about the Vista CTP builds first. In my review of the September 2005 CTP build (5219), I noted numerous feature inclusions such as a dramatically improved Internet Explorer (IE) 7, Media Center Vista, new Tablet PC functionality, premium games, and various new productivity applications, and security features. A month later, I reviewed the October 2005 CTP build (5231), the first to include working versions of Windows Media Player 11, Windows Digital Gallery, Mobility Center, Network Center, and various new IE 7 features. I also complained, somewhat dramatically, about the changes Microsoft made to Media Center, though the company later informed me that it was stepping back from those changes because of complexity complaints, so I'll examine that issue further in my upcoming Windows Vista December CTP review.
Microsoft didn't release a CTP in November 2005. It did, however, release a CTP candidate version--build 5259--to its Technology Adoption Program (TAP) partners, which leaked to the Internet within days. I published a number of build 5259 screenshot galleries (see the links on my Windows Vista activity center for details) but didn't review the build because it wasn't an official release.
In late November, Microsoft held a Web conference with beta testers, and a separate conference call with the press, to explain how the CTP was evolving. During these communications, the company explained that it was accelerating the addition of new features to Windows Vista and would ship both a December CTP and, internally, a feature-complete Windows Vista build by the end of 2005 (see my Microsoft Supplies Update on Windows Vista Beta in WinInfo for details). These changes caused a bit of consternation in certain circles. So let's see what Microsoft has to say about the how the CTP program is going and how it's changed the way the company is developing Windows Vista.
From beta milestones to CTP
With the CTP, Microsoft is moving away from its previous release paradigm, where major product milestones like Beta 1, Beta 2, and various release candidates mark development progress. Instead, with the CTP system, Microsoft will ship nearly monthly builds to testers, introduce features more quickly, and get better feedback as a result. And if the new system works, the resulting product will be higher quality as well.
"The goal of the CTP was to provide a higher frequency of builds to testers," Microsoft lead product manager Greg Sullivan told me during a recent briefing. "There was a lot of discussion about whether we should even do this. There are costs, as well, in terms of impact on the development cycle. [It's hard] getting builds to that [quality] level at that frequency. The feedback you get and the improvements to the process need to outweigh the costs, clearly. And I think the early feedback is that it definitely does. This is clearly something that's been worth doing."
In some ways, one might argue that Microsoft is returning to its roots. On early consumer Windows betas, such as those for Windows 95 and Windows 98, Microsoft often shipped weekly builds to testers. With Windows Vista, the company is returning to a shipment mode in which testers get refreshed code far more regularly than they ever did with any NT-based Windows versions.
Driving to code complete
The most controversial bit, however, was the company's recent announcement that it will ship a feature complete version of Windows Vista--internally--by the end of December. What changed? And why is Microsoft doing this?
"You end up getting a lot of feedback," Sullivan said. "So what do you do with all that feedback? It's a good thing, but it has actually impacted the rest of the process, and the schedule. So that's why we've gone to this notion of accelerating the drive to code complete. It's a core change. The way it was historically [with previous Windows versions], and the way it was with this product previously, is that we mapped it out with a schedule of milestones. You knew that certain features would appear at certain milestones. And so you'd get a raft of features checked in with each milestone, and you'd get as much feedback as you could. Then you iterate and you fix bugs. And then you check in a whole raft of other new features, get feedback, iterate, fix bugs, and so on."
It turns out this wasn't a great way to do things. The challenge in that model is that testers didn't have what amounted to the final product until later in the process because Microsoft was trying to generate feedback about certain features along the way. Now, with the CTP program, the company is getting more feedback and on a more timely basis. "We looked at it and said, OK, we could continue with [the traditional] approach, or we could accelerate and just get more features in, and get to code complete sooner," Sullivan told me. "And then we could use this monthly process to get the large volume of feedback on all of the code so that we can convert to a mode where we're just fixing bugs and not checking in new features."
I do believe that the CTP provides Microsoft with more relevant feedback, because I watched as the betas for Windows 2000 and XP went for several months without any new milestones, leaving testers with nothing to test but code that had been out of date for some time. But the code complete thing. It seems kind of unbelievable that Microsoft could assemble a feature complete version of Vista that quickly. Surely, the Vista beta testers would be a bit leery of such a thing.
Enter the beta testers
Many beta testers believe that their job is to act as unofficial product designers, helping Microsoft determine which features are included in the products they test. After over a decade of beta testing, however, it's clear to me that, with Microsoft at least, most beta testers are just expected to find bugs, file bug reports, and provide feedback about the builds the company provides to them. With Windows Vista, this has never been clearer: Microsoft will deliver a feature-complete build of the product nearly a year before Windows Vista ships, making it obvious that beta testers are simply there to file bug reports. Many of them, I opined, would be upset about this turn of events.
"Hopefully beta testers don't have expectations that they get to kind of vote on features," Sullivan responded. Actually, they do, I told him. "Sure," Sullivan agreed, "but I believe that's somewhat mitigated by the fact that we've done nearly-month builds. That's one of the goals [of monthly builds]: Get feedback on new features earlier in the process."
Beta testers do have a role, I was told. "Even with the acceleration of the code complete date, in the grand scheme of things, that shift is measured in weeks or months at most, and not very many months. So the argument that this [change] gives folks less of a chance to kind of vote on features and give that kind of feedback is not invalid, but it's really just a matter of degree. And it's minor shift, frankly, in the timeline to code complete that's going to result in a higher quality product. This is really optimized for [improving] the quality of the RTM build. So, conceivably, is there some degree of feedback about implementing features differently or whatever, yeah, conceivably there is. We did shorten the timeframe during which we could take that kind of feedback. But weighed against what we think is the improvement in quality of the final build is worth it."
Who are these people?
Every single day, someone writes me an email and asks a variation of the question, how does one get in the beta program? Who is testing? How does one contribute? You might be shocked and saddened to discover how these people are chosen, and how unlikely it is that you will ever get on a beta, regardless of how qualified you may be.
"There are over 625,000 people folks who got Beta 1 and are getting every CTP," Sullivan told me. "There are three pools, or channels, that those people come from. The smallest segment of that population is the tech beta members. Those folks are nominated primarily through our field sales organizations, who work directly with customers in all channels from retail and consumer to small business, to partners, to enterprises. We try to get a cross-section of the user base and we have folks who have signed up to be in the beta program. What that means is that they get more than just the code; they also get phone support and a more direct relationship with Microsoft because the expectation of them is higher as well. [The hope is that] they'll really participate in the process and provide feedback and really kick the product's tires. There were between 10,000 and 20,000 tech beta testers for Beta 1 and in future milestones that number will increase."
The remaining 600,000 or so are members of TechNet and MSDN. These people obviously get a lot of code from Microsoft on a regular basis, so they're familiar with the process. But they form the largest part of the population of people testing Windows Vista, and my guess is they don't contribute much as a whole. I could be wrong, I guess. But it seems like the most active communities for beta testing are simply closed off to most people.
Customer preview program
While Microsoft had previously promised to release Windows Vista Beta 2 to the public via its Web site, the company is now only saying that it will likely release a future build, possibly at future CTP, to the public. That release will happen sometime in the first half of 2006. Sullivan, however, wasn't so specific. "In previous releases, we've also, at some point in the process, as we get closer to RTM, done a customer preview program where anybody, without signing up, can get the code," he said. "So one might assume that we're doing something similar with Windows Vista."
What's coming
OK, so here's where we're at. Microsoft will ship Windows Vista December CTP release this week, a feature complete Vista version by the end of the month, and a feature complete CTP by January or February 2006. Vista Beta 2 is up in the air, but we can look forward to near monthly builds of Windows Vista throughout 2006, I was told.
"We're on track to have Windows Vista on new PCs by next holiday season, which is important to a lot of our partners," Sullivan said. "And of course those are the expectations we set. We also want the quality of this release to be very, very high. So we're optimizing for the latter [quality] and driving towards the former [a late 2006 release date]."
In the short term, there are other Vista milestones to observe. For the Consumer Electronics Show (CES) in Las Vegas in January 2006, Microsoft is working on daily builds that will show a bunch of new functionality that testers haven't seen yet. However, Microsoft doesn't plan to release a feature complete Vista build at CES. Indeed, it's not clear that there will be a January CTP build at all. "We may have put too firm a stake in the ground about committing to delivering something every single month," Sullivan said. "What we meant to characterize this as is 'about every month,' or that kind of frequency. We didn't do a good job of communicating expectations. We knew it wouldn't be every single month."
Regarding the feature complete Windows Vista version that many testers are excited about, Sullivan said that it might be a few months before that turns up in testers' hands. "I think the timeline is that we'll ship a feature complete build of Windows Vista by the end of December [2005] internally," he said, "but this will be after the December CTP." It will be January or February before testers see a feature-complete build. "It's going to take a long time to get some of that stuff working. That's part of the cost of accelerating the code complete date. The next CTP after December will be whenever the code gets to the point where it meets our quality standards. It will be feature complete, but we're giving ourselves a bit of wiggle room to be able to check in something else towards the end of the process. Nothing is absolute. We reserve the right to add a feature later. But we are shooting for code complete this month."
That said, look forward to some cool consumer-oriented Vista features during CES, which I'll be covering here on the SuperSite for Windows. "There will be some stuff we show at CES that will be above and beyond what you see in the December CTP," Sullivan told me. "CES, if you think about it, is the show where people are looking ahead to the next holiday season [2006]. So that's what we're optimizing [our CES presentation of Vista] for. It will really be consumer focused, the first time that you'll see an explicit communication about why mom and dad will want a new PC with Windows Vista or will want to upgrade to Windows Vista."
The CES revelations will be related to Vista's consumer features, such as digital memories, an umbrella term Microsoft uses to refer to digital photos and video. "You can look at the scenarios we've outlined," Sullivan added. "There is a lot around mobility, including synchronization, new PC form factors, pen input, and all kinds of other cool stuff. And the consumer scenarios around pictures, videos, and music, or digital memories in general, that's just such a huge driver of PC sales. The research we've seen shows that digital photography is a huge driver of PC sales and upgrades. It's turning out to be a killer app, in much the same way that spreadsheets and the [Web] browser were. The broad category of what we think of as digital memories, and the sharing and other vectors related to that, is a big area of investment for us. So you'll see some updates there, at CES."
And then there's Media Center. Sullivan told me that Microsoft was "really, really pleased" with the sales momentum of XP Media Center Edition 2005 recently. "We're going to point to that sales data at CES and the state of the current version of Media Center, and say, look, from a PC standpoint, we invented this category," he said. "This is some really significant innovation that is clearly now being broadly embraced in the consumer space and is viewed as another killer app. It's driving the adoption and usage of these PCs and various digital memories scenarios. OK, great. Now, how do we do what we do and make it better, and make the next release better than the last one."
Sullivan also noted that my "frank and candid" criticism about the complexity problems with the Vista version of Media Center in my October CTP review was dead-on, and that the feature had since been dramatically simplified and changed. I'll discuss the Media Center changes more in my upcoming review of the December CTP.
Next: The December CTP
What this is all building up to, of course, is our next Vista milestone, the Windows Vista December 2005 Community Technical Preview release. I will be reviewing that particular build as soon as possible this week. Stay tuned.