With Windows Server 2008 heading toward a first quarter 2008 launch, I sat down with Alex Hinrichs, the Windows Server 2008 project manager, to discuss the development of the product and how changes the company made in this regard will benefit customers greatly going forward. There's no doubt about it: Windows Server 2008 is the most customer-driven version of Windows Server yet, and though it's a major revision that will most surely bring with it some compatibility issues in the short term, the long-term benefits are obvious and unassailable. Here's what I learned about the development of Windows Server 2008.
Hinrichs maneuver
Much of the work around Windows Server occurs in building 43 on the Microsoft campus in Redmond, Washington, a rapidly growing Seattle suburb that's home to much Microsoft-related construction these days. Hinrichs runs the Windows Server ship room and manages the development of what is an increasingly complex product line that hits at almost every level of Microsoft's customers.
For his part, Hinrichs is the hands-on go-to guy for everything related to Windows Server 2008. He sets the schedule, defines the processes, and is the point man for any decisions that need to be pushed up the ladder to his superior, Iain McDonald, or Bill Laing, the general manager in charge of Microsoft's Server Business. (McDonald previously performed Hinrichs's duties on previous versions of Windows Server and the Windows client.)
A 12-year Microsoft veteran, Hinrichs was originally hired as a Windows NT program manager and he worked as the release product manager for all Small Business Server releases from version 4.0 to 2003. "I survived through screwing it up all the way to getting it right," Hinrichs joked. On the heels of SBS 2003 being completed, Hinrichs jumped over to Server, while McDonald coincidentally was taking over all of the Server releases. "We met at that time," Hinrichs said, and a long-term partnership was forged.
Know your role
The most fundamental change in Windows Server development over the past decade has been the move to a roles-based management architecture. Microsoft began working on nascent versions of this architecture as far back as Windows 2000, but it wasn't until Windows Server 2008 that the OS was finally componentized, allowing the management roles inside the product to map both to the underlying architecture and to the product groups working on Server.
"The thing we really hang our hat on is that we've had a clear vision for Windows Server 2008 from the beginning," Hinrichs said. "We talked to our customers and they don't think of the product in terms of product versions, but rather about the server boxes. In their data center and server rooms, they can point to different machines and say, 'that's the domain controller, that's the file server, that's the Web server, and that's DHCP. That?s how they think. The problem is, w we produce a Swiss army knife kind of Server product that does a bunch of different things. But customers wanted to define their servers by the roles they performed." Thus, the roles-based architecture in Windows Server was born.
"This also allows us to better engineer the product from the start," Hinrichs added. "Roles define things from the beginning, and deep componentization means we can install as little functionality as possible by default and give admin only exactly what they need." Even Microsoft's engineering teams, with few exceptions, are organized by roles. "We have general managers and product unit managers who job, literally, is to manage things like the Terminal Services business, the Active Directory business, the IIS business, and so on," Hinrichs said. "We engineer the product soup to nuts to make that happen. Again, it's a very clear focus and vision that helped us scope the product and make the right decisions."
Processing change
From a process perspective, Windows Server 2008 has been in development for several years, and longer than any previous version of Windows Server. To manage that complex of a product over that many years, and not run into the problems that, quite frankly, were rampant with the Windows Vista team, is quite an achievement. In stark contrast to that of Windows Vista, Windows Server 2008 development was steady, sure, and without controversy.
"The way we get to this point is that we had a clear feature list from the start," Hinrichs told me. "We locked down the bulk of the features we wanted to include in 2005, and did a final triage on features back in December 2006. Of course one or two things did trickle in over time, but the feature list was locked and loaded at end of 2006."
By doing that, the Windows Server team was able to hit its milestones as well as a impressive quality bar without having to "churn the code" in response to unexpected feature additions. "We just have so many OS components," he said. "If you make a change to some low level component, components that are high on the stack can be affected. So you have to lock it early. It reduces the amount of shifted sand."
This clear vision of a roles-based product has allowed Microsoft to steer a ship that, frankly, would be unwieldy otherwise. "We can tell a few thousand people [in the Server division] that this is what we are doing, and then they can execute on it," Hinrichs said. "The scope of our customer base is pretty big, from small businesses to the enterprise. But it's not as big a challenge as what you see on the client [i.e. Windows Vista,] where the scope runs from the consumer all the way up to the enterprise desktop."
Learning from the past
Of course, the development of Windows Server 2008 was also affected by lessons Microsoft learned from previous product versions. "What we tried to do with Windows Server 2008 is take the positives from the Windows Server 2003 playbook and replay those," Hinrichs said. "And then we avoid the mistakes. We know what's working because customers tell us it's good."
The most important lesson was to deploy early and often. Microsoft knew that some of the more "disruptive" server roles--things like Web server, Active Directory, Network Access Protection (NAP), Terminal Services Remote Gateway, and so on--needed to be rock solid before they could be shipped in final form to customers. So these roles got at least 18 months of testing internally at Microsoft and externally with its TAP customers. "Over time, we've discovered that that's the magic time period," Hinrichs noted. "So that's what we've done, and over two years in some cases. We deployed Beta 2 [back in 2005] within Microsoft and externally with 50 customers we watch very closely. They get weekly calls and extra funding, and send people out to visit them on the road regularly. They were running [Windows Server 2008's] Active Directory two years ago, just like [Microsoft]."
Feedback from the TAP deployments led to a dramatic improvement in quality as Microsoft fixed bugs related to reliability and usability. "We'd get admins telling us that certain UIs didn't make sense," Hinrichs added. "Eventually we got to the point where the Active Directory role was so stable that every Monday we're updating our Windows development domain with the Monday build. It's in such good shape. But you can only do that with 18 months of deployment to validate that it's ready."
Realistically, Microsoft realizes that even the most strident beta test process won't uncover all bugs because some issues simply don't crop up until you've gotten the product out in the real world. "We can't predict everything," he told me. "So the only way to make sure is to deploy broadly. We built that religion with Windows 2000/2003, and it's the mantra we live by for 2008."
IIS is another good example. Microsoft has been incredibly aggressive deploying IIS internally and externally, and Microsoft.com has been running on this version of IIS for years now. Microsoft also pushed the IIS Go Live program to customers as early as Beta 3. "The message is simple," Hinrichs said: "Deploy, deploy, deploy."
Building the future
Internally, Microsoft has restructured the build process for Windows Server 2008 so that it, like the product itself, is more compartmentalized. There is a Main OS build every day, as with previous product versions, but the process of getting revisions into that Main build is far more granular than before, owing in part to the complexity of the product: There's just no way a single build team can handle all of the changes that are coming in during active development.
"We've maintained the virtual build lab (VBL) structure from previous versions but improved on it," Hinrichs said. "In addition to main daily build, we have several important architectural components that live here the integration points hit. These are Core, networking, server roles, user experience, shell, file system, and a few others."
Under the Under server roles group, for example, you will see sub-groups like the Active Directory team, the Terminal Services team, the IIS team and about 20 others. The developmental lead on the AD team checks in code at the server roles level while inheriting code from above. Once that code is ready for broader consumption, it is checked into a higher branch and consolidated back into the Main development tree. This process is ongoing, obviously, and requires people at each level who can be trusted to monitor the quality and necessity of new code additions.
"There's code flowing up and down the tree nonstop," Hinrichs said, "but we maintain high quality at both levels by a set of quality gates. These gates are looking at BVTs [build verification tests], a battery of tests against sub-builds that make sure something the AD guys do doesn't break other things or prevent other teams from testing. Everything has to work."
Microsoft also runs code quality tools against the code, which looks for security bugs, buffer overruns, and anything else that might cause pool leaks, application compatibility problems, internationalization problems, and so on. There are also code dependency checks--some 40-odd layers of dependencies between components, Hinrichs told me. "To maintain the componentization of the OS, you have to make sure you aren't unwittingly breaking dependencies." The goal of all these tests is to catch these issues far down on the tree so that they affect the smallest possible group of developers.
"All these tools run overnight," Hinrichs said. "And we get status reports in the morning. We can see where different teams are."
Because of the componentization of the development process with Windows Server 2008, the ship room strategy has changed since Windows Server 2003 as well. "It's more evolved now," Hinrichs said. "We don't just have the main ship room. Now we also have seven distributed ship rooms, run by people who meet with the people checking in code below them. They all have daily meetings, as does the main ship room. The main ship room's agenda is simple: Who in the seven distributed ship rooms is ready to bring code up [the tree into the Main build]?"
Since the main ship room is no longer used for triaging code bugs that are now handled lower in the tree, it's emphasis has shipped somewhat. "What we've gone to is more to provide global guidance in the main ship room," Hinrichs said. "We communicate what the focus is, the testing we're doing, but we have to rely on local expertise [lower in the tree]. It's much more distributed now, with more local ownership. The system is just so big. As you can imagine, the people in the middle tier have to work up and down the tree and end up working their butts off. They have over 20 groups below them and me on the top. It's a very, very tough job."
"We do triage many bugs in main ship room," Hinrichs added. "And it's still a big part of the meeting daily meetings, about 30 to 60 minutes each day. We just don't look at 100 percent of the bugs there anymore. Much of that is distributed to the VBL sub-ship-rooms."
Stepping into the daily ship room
"It's real simple," Hinrichs told me. "You walk in each morning and find out what is the state of the main build. We build it overnight. Did the build breaks any of the BVTs [build verification tests] or the thousand or so self-host tests we run? We look at the state or health of the daily build."
Build breaks are very rare, Hinrichs said. In such a case, some errant code from further down the tree actually broke dependencies or functionality in the main build, causing a temporary build cessation while the culprit is found. While this calamity occurred a number of times during the development of the previous Windows Server version, Windows Server 2003, it's only happened once or twice to Windows Server 2008.
"The day to day focus of [the ship room] until fall [2006] was Windows Vista," Hinrichs said. "Here on Server, we kind of rode along for the ride. But once Vista was done, there was a changing of the guard almost immediately, and I took over ship room." Now, he said, Windows Server 2008 and Vista Service Pack 1 (SP1), which are being developed in tandem, are the focus. "Starting from Vista RTM [release to manufacturing, in early November 2006], we flipped the switch pretty quick," he added. "It was like two aircraft carriers passing each other. But working from the Vista RTM code gave us a very stable code base, so we were able to go full bore from the get-go. We have been able to create a stable main build every day since then, since most of the problems are trapped before they even get here."
Entering lock-down
In keeping with their habits of learning from the past, the Windows Server team carried over another bit of wisdom from their experiences on Windows Server 2003: They locked down the code for this product fairly early in development. "We locked down the feature list at end of 2006," Hinrichs said, noting that a change management process was then set up so that anyone who wanted to make a change request would have a formal process to follow. "We have a board consisting of me, Iain [McDonald], and Bill Laing [Iain's boss]," Hinrichs told me. "When someone wants to make a functional change to the product, they meet with us. We weigh the consequences of those changes."
Most changes aren't accepted, as the Server team is leery of feature creep, which it feels created problems on early Windows Server versions. And because of the changes within the organization, developers can't just check in new code on a whim: Microsoft has quality gates in place, up and down the code tree, to see what's coming in both up and down the tree. "We worked hard to maintain the feature list we finalized and have only taken in changes that were either blocking deployments or because the business case was so compelling we simply had to," Hinrichs confided.
This raised an obvious question. Which features were added after the late 2006 functional freeze and which were not? Hinrichs wasn't particularly keen on discussing too many of the changes that were not added to Windows Server 2008 for fairly obvious reasons, but one does stand out: Very late in the development of Windows Server 2008, the team decided that they simply had to add the Windows PowerShell to the product, even though the existence of two scripting and command line environments, one of which (PowerShell) wouldn't be fully supported with built-in management tools for all of the product's capabilities, would be confusing to some customers.
"This is precisely the type of thing we reject," Hinrichs admitted. "You don't want to randomly add some new administrative tool to the product so late in the game, and there's no way to create the necessary PowerShell scripts after Beta 3 [which was when PowerShell was added to Windows Server]. We already had command line tools, and had significantly improved the traditional command line environment in Windows Server 2008. So it was a tradeoff. We're committed to PowerShell, and love it, and we wanted it in the box. But we didn't want to slip the release date just to add some administrative scripts for PowerShell. We're not going to add gravy."
On the flipside, after Microsoft had revealed the Server Core install option for Windows Server 2008, its customers immediately began providing the company with feedback about other roles they'd like to see added. The number one requested role was Web Server. But because Server Core doesn't (yet) support the .NET Framework, there was no way to create a version of the IIS (Internet Information Services) Web server for that environment that supported dynamic capabilities like ASP .NET. The Windows Server team decided instead that it would support a Web Server role in Server Core. But that version of IIS simply won't support any .NET-oriented features in the first release. The result is a low-impact version of IIS that's super-secure and can take on low-end Linux-based servers. Customers were ecstatic.
"The feedback from Beta was pretty clear," Hinrichs said. "Customer told us, 'wow, this Server Core thing is fantastic, I love it, but you missed the most important role. So we went and took care of that."
Adding IIS to Server Core was a big undertaking but it was worth it because it impacted so many customer deployments. PowerShell, by comparison, is something the company is moving towards, but it wasn't absolutely critical to have a complete set of admin scripts in the initial release. It would be nice, but it's not critical. (It's worth noting that Microsoft will indeed support Windows PowerShell with a full suite of administrative scripts via a Web-based library concurrently with the release of Windows Server 2008. And Hinrichs noted that an amazing community of PowerShell users has already sprung up; this group, too, will no doubt supply users with useful and compelling scripts for that environment.)
Working with customers
One of the most amazing changes in the way that Windows Server 2008 was developed is the way that customer feedback has integrated so thoroughly into the process. "The old way of doing this was that customers would file beta bugs, and then we'd have call downs and onsite customer visits," Hinrichs told me. "We still do those things, of course. But on top of that, we can now process CEIP [customer experience improvement program] data, which is information that comes back from the servers that are installed around the world. This information, which is strictly opt-in, is pumped back to Microsoft so we can study it."
It's a considerable amount of data. Microsoft received CEIP data from over 165,000 installations of Windows Server 2008 Beta 3, which shipped in late April 2007 to about 800,000 customers overall. Microsoft has over 1100 server roles deployed on Windows Server 2008 internally and externally, and the company is now building Windows Server 2008 with Windows Server 2008. (Also, Microsoft.com is already running IIS 7.0 in deployment.) Microsoft knows how many customers have installed the product, and even exactly what roles they installed. And yes, I can tell you what those are: Percentage-wise, the most popular Windows Server 2008 roles are Active Directory, Web Server, File Server, and Terminal Services.
Windows Server 2008 also supports the notion of Features, which are functional components that aren't as broadly defined as roles. The most popular roles so far are PowerShell and, surprisingly, Desktop Experience, the latter of which allows the Windows Server 2008 desktop to look more like Windows Vista. Hinrichs told me that people had complained about feature in Windows Server 2003, so they carved it off of the default install so that those who wanted it could install it separately. These people are typically enthusiasts, he said.
The Windows Server team also sees which combinations of roles are getting installed on the same server, and which architecture (x86 or x64) they are installing. "We know that x64 is what all customers are buying now and in the future," Hinrichs said. "The deployments are all x64. There is some testing on 32-bit [x86], but we also know that many of these are in virtual machines [which tend to be 32-bit only]. So we put our testing and development wood internally on x64. The SQM really helps us know that we have the right focus and priorities.
Customer feedback also helped Microsoft determine the feature set for Windows Server 2008. "That was actually the origin of Server Core," Hinrichs told me. "Customers were saying that there are all these services running they don't need, all these extra components. They were installing QFEs [hot fixes] that had nothing to do with server roles they're running. That was the genesis of componentization: Cut the dependencies and arrive at a smaller, more paired down version of the OS. It runs lean and mean and doesn't need a GUI, Internet Explorer, or the .NET Framework." That said, Microsoft is also evaluating changing the supported component mix in Server Core for the future based on customer feedback.
Read-Only Domain Controller is another example of a Windows Server 2008 feature that arose out of customer feedback, this time from its enterprise customers. "They said they loved Active Directory but had branch offices all over the world," Hinrichs said. "Replicating across the WAN was painful. Maybe they had a head office in New York and an oil rig in Khazackstan. They don't want all of their passwords replicated to that remote location. Read-Only Domain Controller caches passwords only for the people how logon in that location. They're not giving up the goods if they get compromised."
Calm before the storm
As a long time Microsoft observer, I noted to Hinrichs that the Windows Server 2008 development process, while lengthy, seemed to lack the drama and disappointments that marred Windows Vista. He wasn't particularly interested in making direct comparisons with the Windows client team, but he did tell me that the calmness exuded by the server team is a direct result of the quality and maturity of the team they have in place.
"It may seem calm on the outside," he said, "but that's because we have a lot of senior people on Windows Server, including some folks who shipped many versions of Server over time. They are very effective at planning and managing their businesses, and they deliver what they promise. The seniority factor really helps."
"The other thing is that Server people are Server people," he added. "We just love working on Server. We haven?t had a hard time retaining people at all."
Hinrichs also credits the team's clear focus on server roles as a factor in the clarity of the product's development. "This isn't BS," he said. "We get a lot of steadiness from Bob [Muglia], Bill [Lang] and Iain [McDonald]. They're steady, and their world is steady. We go to ship room, we set milestones, and we figure out how to work with the teams. We tried hard to be consistent, and maintain a consistent rhythm."
Part of this rhythm involves some of the little things that other organizations simply don't get. Employees aren't asked to work most weekends because the team wants people to see their families and want to be at Microsoft for a long time. "We tried things that way, once," he said. "It didn't work."
The result is indeed a smoother running organization. When Microsoft shipped Windows Server 2008 to almost one million people, with 1000 external real world deployments and 600 internal deployments, the Windows Server team sat back and wondered whether the complaints would come pouring in. It never happened. "The builds are just so solid," Hinrichs told me. "There's just nothing major wrong. We haven't had a million customer problems."
Hinrichs told me that the executive in charge of the TAP program is on call with all of the Fortune 500 companies that are deploying pre-release Windows Server 2008 versions in production. "They have his cell phone number and can call him 24/7," he said. "Remotely or via a plane trip to visit them on-site, he will fix whatever problems they're having. He's only been woken up once in the last six months. That's it. That was not the case with previous Server releases.
One area where Windows Server 2008 has, perhaps, veered a bit off course, however, is the Windows Server Virtualization technologies, codenamed Viridian. This feature is now due in public beta form when Windows Server 2008 ships in Q1 2008 and will appear in final form within 180 days of that date. Hinrichs curiously wasn't interested in taking the easy out on this one, as Viridian was actually being developed outside of the Windows Server team for much of its existence and was only recently pulled into the core OS team.
"It was a parallel project, but we're just glad we have a CTP [Community Technical Preview, which appeared in RC0] so that folks can have an early look," he said. "Some parts of Windows Server 2008 have been stable for a very long time, like IIS, which began a Go Live program before Beta 3. Active Directory is the same thing."
Edited versions of this article originally appeared in the December 2007 and January 2008 issues of Windows IT Pro Magazine.