Windows XP Service Pack 2: The Inside Story

Windows XP Service Pack 2: The Inside Story

SuperSite readers will remember Todd Wanke as the guy who ran Microsoft's War Room for Windows Server 2003 (chronicled in Windows Server 2003: The Road To Gold Part Two: Developing Windows). Todd, you may recall, had pledged to never again run a War Room after the grueling Windows Server 2003 development process. "No way," he said, laughing, when I had asked him then if he would do it again. "No way."

But in mid-2003, Microsoft needed Todd again, this time for what would ultimately prove to be the most secure client product that the company would ship to date: Windows XP Service Pack 2 (SP2). Recognizing how strategically important it was that Microsoft get this release right, he threw caution to the wind and signed on for what he initially envisioned as a three-month project. Over a year later, Todd and a virtual team of Microsofties that worked outside of the usually strict hierarchical system at Microsoft delivered Windows XP SP2 to an eager audience. The SP2 product they shipped bore little resemblance to Microsoft's original plans for the release, but was instead a far more secure and stable product that, ultimately, made XP a better operating system.

In early December, I sat down with Todd, Ryan Burkhardt, and Jon Murchinson to discuss XP SP2 and the virtual team that made it happen. Here is their story.

Windows XP Service Pack 2: The early days

Paul: These stories always have a "Jim Allchin came back from vacation..." kind of start to them, but ...

[Laughter]

Todd: This was the SWI [Secure Windows Initiative] guys. After Windows Server 2003 shipped, I went over to the security group and was sitting on the 6th floor [of building 40], and I happened to be sitting with the SWI guys, which is our security response team. Blaster had just hit, Slammer had just hit, and there were a couple more in the queue.

Paul: A couple of new exploits?

Todd: Yeah, a couple of new exploits in the queue. George Stathakopoulos who runs the SWI team, came into my office and asked what I was doing for the next couple of months.

[Laughter]

Paul: "Couple of months."

Todd: The initial vision of SP2 was not the SP2 that happened.

Paul: Right.

Todd: The initial vision was, we were going to enable the firewall, and we were going to ship it. And with that, I had a conversation with Brian Valentine in about June or July 2003, and then we had a series of executive meetings, with almost every Windows executive in the company. These were very large senior meetings with everyone there, from Jim [Allchin] on down, and there would be 30-40 people in the room and half of them were execs. And we really took a step back and said, OK, what is [SP2]? What are the current problems that we have, and what are the problems we can solve in the short term, mid-term, and long-term?

Todd: The initial vision of "just enable the firewall and ship it" started to get a little bit cloudy. That's because we did some initial application compatibility testing with just enabling the firewall, with the default firewall that was in XP [RTM] and XP SP1 and it just didn't work very well.

Paul: That's what happened on Windows Server as well. Turn on the firewall and watch everything break.

Todd: We knew we had a bigger problem than just enabling the firewall. And so at that point, I sent out a mail to everyone in the division saying, "This is what we're going to do. We're going to take a little bit more time to do it. And if you want to submit a security feature, you should do so, and then show up at this room." Well, the next day, it was standing room only, and everyone had a security feature that they wanted to check in. It went all the way down from things like the new Bluetooth stack, to the new Windows Media Player, to the new Group Policy stuff, and on, and on, and on, and on.

We started walking through this list and realized that this was now not just an SP1 plus, it was now an SP2 [as it was finally released, a major update]. We took the new DCR [Design Change Request] list and Brian and I approved those individual DCRs, and then we pulled SP2 from the Windows Sustained Engineering (WinSE) team.

Paul: And that's not the core [Windows] development team, right? These are the people who typically [develop] service packs?

Todd: Exactly. And it's a much smaller team [than the core Windows development team]. And so we got to that point, and we had our first couple of builds, and then we realized this was a really big project.

The SP2 virtual team

Todd: Brian and I recruited some additional people, and that's when we brought in Ryan [Burkhardt], we brought in Rebecca Norlander, we brought in Laurie Litwack, Richard Ward, and I recruited Mark Harris, and then another gentleman by the name of Tokuro Yamashiro. We assembled a virtual team of about seven people and we just marched forward from that point.

One of the things I do when I run a project is I never use the word "I." Even if you went back through every piece of mail I wrote for Windows Server 2003, and Windows XP SP2, you'll never see the word "I" in any of those emails, unless there was a specific reason for it. I'm just a believer in that if you want to get things done, the best way to do it is as a team. As part of XP SP2, we just assembled a virtual team. And we had done the same thing in Windows Server 2003. Everyone felt like they were a part of [of the project], and could influence it, at an equal level. And whether or not there was a reporting structure, everyone had an equal say and a voice, and I got to break ties. But these seven people were all part of this virtual team, and everyone on that team deserves equal credit for releasing XP SP2. Windows is so big, that everyone had a specialty. I guess I'm good at running projects and getting people to [see] the finish line. Ryan is really deep and knowledgeable in setup and base. Richard is really knowledgeable in architecture and so forth. Rebecca is really good at managing and communicating to the execs. So everybody had a different specialty, and everyone had a key role in the team. And that's how we brought it all together.

Ryan: Rebecca, Richard, Laurie and I kind of did the ... technological program management. We kind of divvied up how the different security features would interact. Laurie was user experience and made sure that when we came up with NX [No eXecute] and the DEP [Data Execution Protection] support, it was usable. I would make sure the Firewall worked well with the RPC and DCOM changes, and the NX stuff. Todd and Mark drove the ship and got it out the door.

Todd: I think there's an interesting story around the virtual team. There's obviously got to be a point where there's just one person, who is the face or the butt, depending on how you look at it.

[Laughter]

Todd: Sometimes I happen to be both. What was happening behind the scenes is a virtual team, a great team, I think one of the strongest teams in Microsoft's history.

Jon: Yeah, as someone who came into the team in May, as I was saying, the development work had obviously gone on before that. A lot of the PR work needed to be done. Starting there and leading up to RTM. Ryan had mentioned that he had gone out on tour earlier this year. So there was a lot of PR work that was done, but a lot of the work we did was actually after RTM, working with people like yourself, working with the field, providing messaging and guidance and the like, and a lot of it was fielding questions from PR managers and the subs, or from leads and subs. Todd mentioned the virtual team and people that you've never met before, or people that are in far-flug locales who are asking questions or seeking guidance on some aspect of it, I'm sure these guys have found as well. It's certainly something that the PR team was pretty involved in. It's something I found personally enjoyable. You had to ramp up very quickly. There was very little time, but people were very helpful. The number of times you'd get Ryan on the phone, either doing this local radio in Philadelphia for his home market, or whatever the case may be, Rebecca, or people from the security business unit, Rich Kaplan, I mean there were just a variety of people who pitched in and did a lot of really helpful heavy lifting for us.

Todd: Every six months or so I try to travel. I'm not a huge fan of traveling. But I went over to Asia and met with our Asian partners, and spoke with our Asian field, and subsidiaries as well. And there's a team that's just associated with that as well, and they took me, or they took Ryan, to these different subs, so throughout this process there's so much that goes on that kind of builds that network. Now whether that's visiting five countries in three days, or ....

[Laughter]

Todd: [To Ryan] You've done it, right? We just kind of did whatever it took and divided and conquered, I guess.

Jon: And I think people were pretty unselfish too. They did things that went beyond their job description. There's just stuff that needs to be done, but there wasn't time to quibble about it and say, you know, that's not really in my box. People just lent a hand and got it done. Not that people don't do that every day. But it was very focused.

The SP2 timeline

Paul: So was SP2 [as we know it] "there" at that point?

Todd: No.

Ryan: I would say no. At that point, we had the laundry list, and we had a good idea of what we could do. And there began the vetting process of the SWI team, and the other folks, to say, what would be super-beneficial to customers and be shippable [in a reasonable time frame]? Looking at it, it was kind of an elastic band of firewall to the kitchen sink of security to something usable by customers and shippable.

Todd: We really had this kind of time line that started at ... Let's call this Todd Wanke's office. [Todd starts drawing SP2 development timeline on whiteboard; see below.] People would come to my office and say, we want to do something. So then we had this period where we [were going to just enable the firewall].

[click for a larger image]

Paul: I assume that [just enabling the firewall] broke everything?

Todd: Yeah.

[Laughter]

Todd: Jim [Allchin] said, we're going to do it big, and we're going to do it once. If we're going to break everything, let's break everything once, but let's fix the problem ... And then, on top of this, you have these DCRs. And then, after the DCRs, you have all the SP2 QFEs and fixes. And then we need to ship. And so, I'd say from this point to here, it was roughly August to September, and then this point here, let's just call it 10 months.

Paul: It wasn't originally supposed to be that long, was it?

Todd: It was originally supposed to be a three month project. If you think about here [points to the whiteboard]--let's just call this "beta"--and then we had our RC1, things like the Security Center didn't come into SP2 until RC1.

Paul: Right, I remember a lot of things changed late [in the development cycle]. The [DEP] stuff changed during RC1...

Todd: Right. And we really made a lot of changes even up through RC2...

Paul: Which is really unusual, right?

Todd: Yep. It is. This was a big ... I'll give you the inside scoop. This was a big fight. Calling this RC1 and not [a beta release]. The reason we called it RC1 was that we wanted people to think that we were serious.

Paul: This is the point where people are actually going to start rolling this thing out, even publicly.

Todd: Yes. We called it [a release candidate] because we really wanted to bring attention outside [of Microsoft], specifically to our antivirus vendors, that we were serious about what we were doing, and we were getting close to shipping ... And so we called it RC1. From this point forward, I would say that the participation in the industry [went up dramatically]. This was probably one of the best things we ever did for SP2.

Ryan: Previous to that we did press events, we did RSA, tours, we kind of pulled all the levers we could, and kind of wave the flag, but the seriousness and real impact of what we were doing was [lost on the ISVs].

Paul: I think the press did a decent job of saying, hey, this is going to be big. I saw that everywhere. You gotta pay attention to this, you know.

Todd: We didn't have ISVs onboard until RC1, or after RC1 I should say.

Making SP2 happen

Todd: The [original SP2] thing was a team of about 15 people. And then, with the DCRs, there were [quite a number of them].

Paul: Was there anything notable that was brought up for potential inclusion in SP2 that didn't make it?

Todd: I can't say that anything was dropped. The biggest thing that was added was the changes to extensibility in Internet Explorer. When we started, with the firewall, things started to change in this period, and we started to take a serious look at IE. And so [it's hard to understate] the importance of the effort that we put into making sure that IE was a much more secure platform.

Paul: When you say extensible are you referring to the APIs that were created for IE, that were new [to SP2]?

Todd: No, Group Policy. Internet Explorer, up until SP2, was not fully Group Policy-aware. There's another that's right here [points at whiteboard] ... this is where we codenamed it Springboard.

Glenn and I sat down for coffee and came up with that name [Springboard].

So I'd say IE was the big one, but also the yellow Information Bar in IE, that was something that, in the beginning, we never thought about. All we were talking about was hardening IE.

Paul: How does IE in SP2 compare to IE in Windows Server 2003? Was the original idea to make it more like "IE Hard"? Obviously, it can't be that locked down.

Todd: The original idea was to make it sort of like IE Hard. The IE in Windows Server 2003 is really unusable for consumers. But we were thinking that drastic at first. I can tell you that during the [initial design] phase were definitely thinking as drastic as that.

Ryan: I mean, throughout the project, that was one of the things that bugged all of us, especially the seven people who were balancing the [overall SP2] schedule, technology, and usability. The team was forever going back and forth on that question. How secure do I have to make it? I can make it so secure that it doesn't work, or I can have 100 percent compatibility. I think it was that initially guidance of the [original SP2] idea that made us head more toward security than compatibility. It did iterate over time, as we got the beta feedback, we got the RC1 feedback. We said, hey, the Information Bar would be valuable, and is in line with [the way certain security features work] in Outlook, and some of the other experiences. Or the NX support, that that should be with the Windows binaries only by default. Because we have to have a balance of security and functionality within the OS.

Paul: At least in that case, you're guaranteeing that your stuff is going to work, and that other people can be ... embarrassed by it.

Todd: I can tell you that throughout the project, especially with interim milestones, we had some periods when application compatibility was near zero. We took some radical steps and said, OK, this is definitely [breaking apps]. Now, how can we tweak that ... and preserve some form of application compatibility?

Paul: Did you end up having to do a lot of per-app ...

Todd: Shims?

Paul: Yeah.

Todd: There are a couple of things I hate to do. One is shortcuts and hacks. Every time someone would bring in an app shim, we would say, let's take a step back and look at this. Are you just shimming that application, or are you shimming a symptom? Let's look at a core fix, or a core change, or a Group Policy setting, or something other than an app-by-app fix.

Ryan: If you don't look at it Todd's way, the platform's scale just screws us. If you do that one shim for just one app, there are probably in the ballpark of ten other apps or more that you're missing. And you're not going to know what they are. It could be a custom application in France, or Singapore. We're not getting visibility into [all the apps out there], so you have to fix it the right way.

Todd: The other thing that I think people don't necessarily do a good job of thinking about initially is line of business [LOB] applications. And that's the other reason you have to be so cognizant of making sure you don't fix one-off problems, but that you actually fix the symptoms. Because there's an extensive number of applications in the line of business space that you will never get a chance to test.

Paul: OK, so this sounds difficult.

Todd: This was a tough release for us.

"Lipstick on a chicken"

Todd: You know I actually have a t-shirt for you ... It has a chicken-- [it says] "lipstick on a chicken"--Darrin Muir said that. But it was for bringing in a shim or something like that. Darrin said, That's like putting red lipstick on a chicken.

[Laughter]

We only made ... 50 of them. The t-shirt is relevant to this because it has the "Top Ten" sayings of Windows XP SP2 on it. And so ... throughout the development process, there are all kinds of things that you say tongue in cheek, or you mean one way but it comes out another, and so we listed some of the better ones on the back of a shirt. One of them, I said, "if we take out all of the failures, what's our success rate?"

[Laughter]

Todd: It made perfect sense when I said it.

Paul: That'd be 100 percent, Todd.

Todd: Yeah ... That was one, and then there was one time I said, "I'm not the smartest guy, but I should be able to figure this out."

[Laughter]

Todd: And there's a list of those things. And they're not all from me, but ....

Ryan: They're mostly from Todd.

[Laughter]

Todd: I was the butt of some good jokes. But it was such a stressful project just because we were in such a state of unknown. There was a lot of passion put into the project to get it to a state of ... sanity. The security problems we were up against were just so broad. We weren't necessarily sure of the scope. And it was kind of in this period here [points to whiteboard at Beta/RC1 phase] that we said, let's start whittling down the solutions we have for the problems we know and let's see what we missed. That was a good point in the project [for us] and I think that was kind of the turning point. OK, this is going to work.

Todd: But as you know, during that period of time, the vulnerabilities we had coming in were just ... they were coming in fast, and they were getting more and more dangerous.

Todd: Paul: I do remember several times, even up until the summer [of 2004] that people would comment on a vulnerability and say, hey, if SP2 were just out, nobody would have a problem with this.

Taking responsibility for the features in SP2

Todd: You know, there were some small things that I really took to heart during SP2. I hated the fact that Internet Explorer didn't have a pop-up blocker. To me, it made pre-SP2 browsing on the Web just a nuisance. Personally, I embraced the things we did in IE as very serious and making sure we did some things there that brought some sense of control back to the user. And so that was a feature I really loved.

I also loved some of the stuff we did around Bluetooth, and I'm a huge advocate for the mobile wave that either has or will soon begin to take off. It seems like everyone is doing Bluetooth devices these days. Whether it's a BMW or Mercedes, or a Nokia phone, or whatever, everything is moving that way. And so the Bluetooth stuff we did, I thought was kind of cool.

The stuff that we did around the player, Windows Media Player. It would have been great to have gotten Windows Media Player 10 in, but we're happy with what we have now.

Paul: OK, so some people look at this stuff and they say, Oh, this is just Microsoft using some excuse to get the latest version of its software in there. But it's not really about that, right? I know that WMP 9 has some security enhancements that aren't in previous versions of the player.

Todd: Right. If you want to look at specific examples of why we did the things we did, let's start with Windows Media Player 9. We could have gone back and done a security push on [WMP 8], which was in XP and XP SP1. But the problem was that we had already done a security audit on ... [WMP] 9. So what shipped in Windows Server 2003 was Windows Media Player 9, which was the new secure media player [at that time]. And what we didn't want to do was go back and do that work on [WMP 8], because we had already done it on [WMP] 9. So during SP2, we invested in further work to ensure that [WMP] 9 was a more secure platform. And that's why we brought [WMP] 9 into the operating system.

So that's one example. Another example I like to give is Windows Movie Maker. People ask, why in the heck did you put the new Movie Maker in [SP2]? And there was one reason, honestly. The old version couldn't be Group Policy enabled. And so we brought the new Windows Movie Maker 2 in, and the only reason I can honestly tell you, and Brian [Valentine] and I agree to it, was they said they could not make Movie Maker 1 Group Policy enabled. Or, specifically, the ability to disable it. Corporate customers kept telling us they wanted it to be disabled.

So those were some of the tradeoffs during this DCR review, where we asked ourselves the question: OK, if we're going to rev an application, what is the reason? And really, security was the reason. You had to have a reason and that reason had to be backed by security.

Bluetooth was another example. Besides the [previous] Bluetooth stack not being particularly good, the new Bluetooth stack a) worked, and b) was secure.

When I brought Mark Harris on board, I kind of mentored him throughout the project. I invested a lot of time personally in him. And one of the first things I said to him was that there are a few things he needed to do when you running a project. First, make sure you're always having fun. Second, I told him he needed to get a hobby, as part of the project. And there had to be something compelling about what he was working on, that he could really get off on. [On SP2,] his stuff was [Automatic Updates]. He had been the Group Policy test manager [for that]. The thing that he got excited about and really invested a lot of his time in was making sure the AU and SUS stuff was up to par.

For me, it was Internet Explorer, Bluetooth, and Windows Media Player. Those were the things that I self-hosted and I tested and I filed bugs for. And I yelled at people when it didn't work right. Those were just some of the things that I did, and felt were important. You have to take a stake. As the project manager, it's very important that you know the project inside and out, and that you're using it every single day.

Will it ever ship?

Paul: How do you compare [SP2] to [Windows Server] 2003? I know that was stressful.

Todd: You know, it's funny, I was actually thinking about this just yesterday and ... I'm still recovering.

[Laughter]

Todd: ... both mentally and physically.

When I say this was a harder project [than Windows Server 2003] ... Windows Server was a hard project because we had the security push during it. And, as you know, during the 2003 project, we actually had a period of time where I thought we'd never ship it. I never got to the point where I thought we'd never ship XP SP2. I just wasn't sure if we could get to the quality level that we need to be at in time.

Paul: In time ... ?

Todd: Before the next [round of dangerous exploits].

Ryan: You said you'd thought that this would always ship, but I personally thought this might not ever ship.

Paul: Well, the amount of time that you slip has to be multiplied by the number of people you expect to use it.

Todd: Yeah.

Paul: I mean, the thing with a server slip is that customers say, fine, get it right. That's what these people want. Unfortunately, when you're doing security stuff on the client, there are a lot of people saying, Hello? We need this now.

Jon: But at the same time, I think they still want us to do it right. It's a balance of urgency and correctness. It's a tough marriage.

Todd: Another thing is, when you get down toward the [final] release [of SP2], there's a whole different set of problems that comes into the equation. Now how are you going to push this thing out? And how are you going to fix those problems that come in [at that late stage], especially if it's a problem that affects, say, maybe one percent or two percent of your users? So as we were ramping down the project, we were thinking about how we were going to push it out via [Automatic Updates] and around the new [Windows Update Services] 2.0 infrastructure that we put out. And we also had to ramp up all of our partners, and all the other things you do for a big release. But this was a different release in the end, in that we weren't going to make it optional [for users]. We were going to slipstream OEMs [PC makers], and so that was a different problem [we needed to address].

Paul: When you finally get something like that out, there's always the fear that some huge class of customers won't be able to install it for some reason, that there's something you didn't [know about], but it doesn't seem like that's really happened.

Todd: There are two sets of problems. In the first type, when you ship, there's a problem that you didn't discover [during the beta] that affects a large number of users. The second problem you have is that the day you ship, the hackers have an exploit and penetrate all of the work you've done for the last 12 or 14 months. And I think those were the two biggest things we were kind of concerned about.

Paul: But neither one of those came to be, right?

Todd: There was one.

Ryan: TV Media. It would actually crash the machine during an SP2 upgrade. It would blue screen, because it modified the files that were going to get updated. It was amazing. SP2 was the largest beta program we've ever had, and this was the broadest release we've had for anything, even [a core] Windows [release], and still we missed ... we just didn't have fine coverage. It's not that we missed it. We just didn't have people cover these kinds of scenarios. It's interesting that our beta testers don't have as much spyware on their systems as other folks.

Paul: I'm sure that they tend to be a little more ... technical ... or on the ball [than the general public].

Todd: Another interesting point about that TV Media problem is that that problem is not an SP2 problem. That was an installer problem. It's a problem that's also shared with NT 4 and Windows 2000, and in XP. So it just so happened [that it came up when SP2 launched].

Paul: Sure. But to people, it happened when they installed SP2. That's what they see.

Todd: Yeah. So those were some of the big things we were thinking about as we got down to the end.

Ryan: Right, and that's why I said that I thought we'd never ship. Because we don't ship unless I know there's nothing coming in the queue that impacts the product and, for a while in the summer, and before, it was, oh, here's a new one coming in, and you have to go through the process of evaluating it, see what's happening, and if it impacts SP2. As we kept iterating through, things weren't that good. It was a risk. And we got lucky in the end that we had a big window of time where that didn't happen.

Todd: The last thing we wanted to do was ... part of the release criteria that we had was that they day that we shipped, we wanted to ensure that when you went to Automatic Updates the day after we shipped, there wasn't a new vulnerability. And so you get into this chicken and egg problem because when a vulnerability comes in, you need to get a fix, and you know how the whole escrow cycle works, you essentially reset the escrow cycle each time. You can get into this ten-day slip over and over and over again, until you go ten days without a vulnerability. But the product held up at the end.

Post-release success

Paul: And since then, it's also been very [successful], it seems ...

Todd: It has, yeah.

Paul: ... surprisingly?

Todd: I think it's met or exceeded my expectations.

Ryan: Yeah. You can never have the expectation that you won't have any problems. I think it's unfortunate that a lot of people, in the beginning, understood that this was a step, and then suddenly it became a panacea for security. Now, I think people understand that it's a step, it's a journey we're going on. And so far so good.

Jon: If you talk to our PSS [Product Support Specialist] folks, you'll find that the volume of calls are down and are lower than we had expected. Many of the calls have not been about issues people have been having. They're questions about how to do things around SP2, or how to get it, things of that nature. Some of them were basic, but still related, obviously, and I think that holds true for our OEM partners as well.

Todd: That's a great point. As we were ramping the product down, PSS was staffing up and was outsourcing for partners, and doing all the things associated with a big release. And we were anticipating just a humongous spike in calls, and it just wasn't there. I hate to say it was a fraction of what we expected, but ... it was in the fractions. It just wasn't there. If we were anticipating 1000 calls, it might have been 10. It was really, really low.

Paul: I think with SP2, more so than with other releases, there was a lot of warning.

Todd: Yeah.

Paul: There were a lot of stories about it, a lot of ...

Todd: Yeah, you guys did a great job.

Paul: Well, I'm not patting myself on the back. I don't mean it like that. I mean it seems like there was a lot of information about there and people were sort of prepped for it.

Jon: [To Todd] I think some of the outreach that you mentioned to the ISVs, I'm going to guess from your timeline that it was about ... what, October?

Todd: For RC1? I think that was in the summer.

Ryan: Yeah, it was early summer. June.

Jon: I know that Barry Goffe went out and talked to a bunch of ISVs last year, last fall, and tried to level-set with them and kind of get them to really test this against their applications. And we did the same thing with our corporate customers about the line of business applications [Todd] mentioned earlier.

Ryan: The TAP program, or RDP, or whatever they're calling it these days. We invested quite a bit in those customer programs to make sure folks got it and were onboard and providing feedback.

Todd: There were over one million installations of pre-release versions of SP2 prior to shipping. We had pretty good coverage.

Paul: So how's the rollout gone so far?

Todd: It's been good. We did an enterprise/IT road show where we went to, I believe, 80 cities, [in Fall 2004], right after SP2 shipped, and we invited our large corporate customers as well as some of our medium customers, and we specifically asked them to send one to two of their IT folks. It was a two-day training [event], depth training on what we did in SP2 and how you can configure SP2 using all of the various Group Policy settings ... so you can deploy SP2 a secure environment. I went to three of them, spoke at three of them to the crowd, and it was very well received. The training went over very well. We ask the participants at the end if they were planning on deploying SP2 in one to three months, three to six months, or six months or greater, and it was a pretty even split between the three different categories. But there was an urgency to deploy in six months or less, rather than in six months or greater. I think for an enterprise or large organization to do anything like that in six months is pretty commendable.

Everyone understood SP2 really well and I think everyone was really engaged at the corporate level. So I don't know. It would be interesting to know what our exposure level is today in the consumer and enterprise spaces. But I would say that, within a year, we'll see a really significant portion of PC desktops running SP2, both within the consumer and enterprise spaces.

Jon: I think the latest number we have around deployments is over 130 million, and that's mostly going to be on the consumer side, or consumers and small businesses. We don't really have a way to track enterprise deployments at this point. Those are all [Windows Update and Automatic Updates] downloads. That doesn't include folks who go [to microsoft.com] and pull down the download themselves, or people who get the CD, or people who buy new [PCs with SP2 preinstalled].

Paul: As far as new boxes go, right now, it's going to be all SP2?

Todd: All of the OEMs are required to slipstream SP2 within 90 days after we delivered SP2. [That time period had just elapsed at the time of this discussion.]

Internal assessment

Todd: From day one, I'd say, from the first build on, if you looked at every computer I owned, I ran SP2 on every single machine. Just recently, I finally got my act back together and finally have stable systems. It feels like this is the first time I've actually had a stable operating system environment where I can actually come in and feel like things are going to work. I don't have to file bugs to get things fixed.

Paul: That will get old after a while.

Todd: Yeah, I know. I went to Frys and bought a 200 GB drive and copied everything I owned over to it, and formatted all my machines. It feels like it's been...

Paul: SP2 just feels like a whole OS.

Jon: Well, the scope of it, in terms of touching so many different parts of the company, is impressive. It's also something that has a number of people invested in it who want to make sure it works well. Obviously, on the dev team, people in the field, people who are doing OEM relations, ISV relations, or government relations, for that matter, across the board, there are people who are all growing together on this, and that was, I think, one of the rewarding parts for me. And I certainly didn't work on it nearly as long as these two gentlemen [Ryan and Todd].

Todd: This was a big win for us. I don't know if we took as much credit as we should have for such a big win. Maybe it's because we had dug our own hole.

Paul: Sure. You don't want to tout how great it works because obviously you're setting yourself up for the next exploit.

Todd: Yeah.

Paul: "Mission accomplished."

[Laughter]

Todd: Things you don't say after you ship a product....

I still get email. I just got one in the last couple of days where some guy said, you know, every day for the last X number of months, I've had to reboot my SP1 system in the morning prior to working, in order to have a reliable [system]. And then he showed his system uptime for XP SP2, and he's been running it for 32 days without a reboot. He said, you guys just did a great job. It's cool to see that that feedback is still coming in.

We really focused on security a lot in SP2, but we also took a lot of robust fixes as well.

Ryan: We raised the quality level. But even performance came up, which is kind of surprising when you look at all the churn we took for security. The performance team worked like crazy to make that happen, and the reliability teams, and I think it was the first time in a service pack where we had numbers that were better than with the previous service pack going out the door. It was unbelievable.

Todd: The other thing that was really amazing about this release was the camaraderie that was built inside of the company. It was something that I don't think we truly understood until the end. But when you think about the things we did, especially during this phase, the [initial] phase, we had about 15 people, a lot of them from [Microsoft] Research. And those people kind of brainstormed ideas. And then we--during the DCR phase--we essentially went to all the groups and asked, what are all the things that you want to do or can do. And then we went to WinSE and essentially ripped SP2 right out of their hands.

Paul: How did they feel about that?

Todd: That was the most difficult part of the project. We had had to do the same thing with XP SP1, but then we did it for legal and compliance reasons. When we lost that injunction, we had to ship SP1 in 60 days or 90 days, [so we pulled it back into the core Windows development team. That was the first time we had ever done that.]

The WinSE team had been working on SP2 for, I believe, three to six months, and they were all excited about shipping something cool. I essentially had to go in there and rip it from their hands. It was really tough, all the way up and down [that organization]. But Brian and Jim did a good job, they said, no, Todd's the guy, and we're sticking behind him, and we're going to run it from core and this is how it's going to be done.

And so that was another challenge. But at the very end, we had the guy who ran WinSE actually sitting next to Ryan and I Wednesday at a recruiting event, and he said, hey, do you guys have any jobs open over there? That's the camaraderie we've built. It turned out to be just a great, dynamic team.

Looking ahead to XP SP3, Windows Server 2003 SP1, and Todd's future

Paul: So SP2 was successful. But what comes next? Is SP3 going to be a more traditional service pack or does it add features like SP2?

Todd: I can tell you the current plan. Obviously, this could change. SP3 for Windows XP is slated to be a service pack the way we've always done service packs, and not to be a major upgrade. It will focus specifically on QFEs and fixes [to bugs] that are reported by customers. It should not contain new features. And customers have been yelling at us for the past few years about that.

Kevin Johnson is the person, I believe who said we'd never do that [ship new features in service packs]. He runs the sales channel. It was either Kevin or Orlando Alaya, one of the two.

So the focus right now is that SP3 will be a regular customer fix and also anything that we might find that's security related as part of our regular investigations. The focus right now is on Windows Server 2003 SP1, making sure that the work we've done in XP SP2 is forward-ported into Server. And so there's a ton of work that's being done there. A lot of the work we did in XP SP2 was work from the security push [that took place during the development of Windows Server 2003] and so I wouldn't necessarily characterize it as ... I think more went from Windows Server 2003 into XP SP2 than vice versa, mostly because of the security push. So right now, Server 2003 SP1 is the big investment we're making.

Paul: So is that some of the stuff that you're working on personally now?

Todd: No, I'm not working on Server SP1.

Paul: You're not doing anything now, are you Todd?

[Laughter]

Todd: Did I tell you that SP2 was a hard project yet?

[Laughter]

Paul: Just start shaking occasionally and people will leave you alone.

[Laughter]

Todd: I moved over to the security team. We're going to focus on the security problems.

Paul: Is that security problems across the board, or...

Todd: No, not across the board. Microsoft has formalized, for all intents and purposes, a Security Business Unit. Before, with security, each of the individual teams were responsible. And we still have that today, but there's more of an effort now to have a specific business unit focus. So there's some teams that are looking at some of the big [issues] today that we all know about, spam, pop-up blocking, the things we're doing in IE [Internet Explorer].

Paul: So this is client and server?

Todd: It is client and server, yes. Right now, I've moved over, as part of that team, in a generic role.

[Laughter]

Ryan: Which can be a good thing.

Paul: Is [Windows Server 2003] SP1 being done in core as well?

Todd: It's also being done by the core team.

Paul: So have you rethought how you do these things, or are these exceptions for various reasons?

Todd: XP SP3 is being done by WinSE. They're already working on it. They day we shipped SP2, they started SP3. WinSE is deeply involved because they own the updater and the installer parts of the technology. So the day we ship SP1, SE inherits it.

On to Longhorn...

Todd: So SP1 is the big investment right now and we realize we have a lot of work to do to improve Internet Explorer, and so there's still a continuous investment in Internet Explorer, and we're just doing everything we can do ensure that we're securing that platform. So SP1, IE, and then we're really getting geared up and focused on getting ready for Longhorn.

Paul: Now is this the Security Business Unit you're talking about?

Todd: I'm talking Windows [Division] in general, or Microsoft in general. The Longhorn wave ... we kind of took a year off. We kind of stopped the train, went back and fixed some problems in XP, and now we're gearing the momentum back up. We are getting ready to focus on Longhorn. So those are kind of the areas we're focusing on right now. You should not anticipate that we will do anything like we just did in SP2.

Paul: Right, but you would certainly not run it again if they did?

[Laughter]

Todd: Absolutely not. No, I'm not sure what I'm going to do next, but I won't do this again.

Ryan: Until the next time.

Todd: Until the next time.

Paul: You're going to get tapped for this again, I can already tell.

[Laughter]

Todd: Yeah....

[Laughter]

Paul: It's coming. Longhorn will slip a little bit... You're going to be like McArthur, you have to return.

[Laughter]

Jon: You gotta be careful though, look what happened to McArthur.

Paul: Right, you can't try to nuke China, Todd.

[Laughter]

Todd: [SP2] was a great project. It was nice to see the company come around, and it just raises the bar for Longhorn.