Based on an interview with one of the people that first discovered the recent "51 IP" bug in Windows 2000, and email discussions with various people at Microsoft Corporation, it has become clear that the company knew about this bug before the release of Windows 2000, did nothing to fix it, and then ignored customer complaints when they began occurring in late March. What's amazing here, of course, isn't the bug itself--which requires a fairly uncommon set up, to be sure--but Microsoft's response to the customers that complained, some of whom are ISPs and other large Windows accounts. But this bug is most likely to affect small companies, a market that Microsoft has historically championed. In this case, it's clear that the company has let them down.
"The problem isn't the bug--the product's only a few months old--this kind of thing is going to happen. I understand that," says Brian Bergin, the president of Terabyte Computers, a Microsoft Solution Provider. "The problem is the response from Microsoft. It's really about the way the bug was handled. This could have been potentially devastating. If you don't \[run into this bug\], Windows 2000 Server is fine. If there's a limitation here no one found, what other limitations are we going to find? But Microsoft was aware of this. This is negligence on their part: They knew about this problem and didn't fix it."
Indeed, Microsoft did know about the bug. In correspondence with various Microsoft engineers, I've discovered that the bug--where only 51 IP addresses can be added to a Windows 2000 Server that is used as a domain controller---was known before Windows 2000 went gold in December 1999. Win2K Server, I'm told, was tested successfully with over 4000 IP addresses on machines that are not used as domain controllers. But it dies after 51 IPs are added when the server is configured as a domain controller.
Microsoft's response to this problem has been atypically arrogant for a company that has been watching itself very carefully during a grueling anti-trust trial. "Frankly, hosting this many sites on a \[domain controller\] is stupid," was a typical response to my queries. And yet Microsoft itself offers a product--BackOffice Small Business Server--that allows you to add up to 50 IP addresses to a single domain controller, a machine that is typically also running Exchange Server and SQL Server, two resource-hungry products. And previous versions of Windows NT and BackOffice were not limited to 51 IP addresses. Are either of these setups "stupid"? Maybe, but that's not the point: The 51 IP address limitation is artificial; it's a bug. In other words, the product is not intended to operate that way.
Most importantly, there are real customer situations where you'd want to run such a setup. Bergin's example is atypical, perhaps, but real: A small hosting company, Terabyte provides asset management solutions for automotive distributors who were previously using a DOS-based Btrieve system to FTP information to a central system each night. Terabyte set them up with a more modern NT-based solution with an automated, custom FTP client. Terabyte has three powerful SMP machines--one for SQL Server, one for Exchange (currently NTMail), and one for IIS. To simplify user administration--clients would require many logons--Terabyte set up a Windows NT domain (and Exchange requires a domain as well). So the question then came down to which machines would be set up as domain controllers, machines that would store the information for authenticated users. Microsoft recommends using at two domain controllers on any network, and since the company couldn't afford to simply buy two more machines, it made the IIS server the primary domain controller (PDC) and the mail server the backup domain controller (BDC). In early 2000, the machines were upgraded to Windows 2000, making the Web and mail servers simple domain controllers, since Win2K does away with the primary/backup controller system. New IPs were added on a regular basis as clients were added, and Bergin estimates that he'll hit the 51 IP ceiling very soon.
And here's the thing: In Terabyte's scenario, the only users that are authenticating against the domain are automatically accessing an FTP site at night and then logging off, so the 51 IP issue is a real limitation. It's just a unique way to use the system, one that perhaps Microsoft itself wasn't aware of. But that doesn't make it any less of a problem. And there are other scenarios--Internet Service Providers, for example--where this limitation should have been far more obvious.
Bergin began looking for answers on the Microsoft Select support groups. Microsoft's first response was that it must be a resource problem, though that person never attempted to reproduce the bug. In some cases, Bergin's questions simply went unanswered. Finally, late at night on March 24th, Microsoft responded with an unrelated knowledge base article; when users questioned this, several more posts went unanswered.
After a series of phone calls with Microsoft's critical support line, the company finally admitted that it had reproduced the bug on March 29th. Microsoft told Bergin that they had never tested Windows 2000 Server in this scenario, though I've discovered that not to be the case (I suspect the critical support people, based in North Carolina, were simply unaware that the Windows 2000 team knew about this issue). Needless to say, Microsoft recommended that Bergin upgrade his network with a new server.
"What does Microsoft expect?" Bergin asks. "People can't just upgrade when they hit 51 IPs. Adding multiple network cards does not work either: The limitation is per server, not per card."
Meanwhile, Microsoft is working on a hot-fix for this issue which should be made available some time in the next few weeks. This is contrary to Microsoft's first public statements about the bug (which it then referred to as an "issue"), when a spokesperson misrepresented the issue on several levels: "Microsoft would not likely produce a hot-fix for this, given that none of our customers have reported the issue," a Microsoft spokesperson told BugNet well after customers had indeed reported it. "If a customer does report this, however, we will take it very seriously." I've since discovered that the hot-fix was already in the works at the time this statement was made.
"Look, I'm a small company," Bergin says. "Microsoft doesn't understand my situation. Legally, I have four employees. I cannot afford to do what the nationwide service providers can do... I can't just add another box. I'm not going to shut down my business because Microsoft doesn’t understand me. They need to respond to actual customer complaints.
