The BIG Announcement: Content Scalability Support Changes for SharePoint 2010

Several weeks ago, a major Fortune 500 organization was considering a SharePoint project, behind which was 50TB of content. Microsoft consultants—along with others in the community at large—convinced the customer that their project simply would not be supported, because content databases would be greater than the 200GB “limit.” The customer turned to Documentum as a solution—a potentially huge loss for Microsoft, for SharePoint, and for the community. Similar stories have been repeated over and over. But the reality is that many of us consultants have supported customers with enormous content repositories and, yes, even large content databases. And ISVs report customer success stories with terabytes of data.

The problem was, simply, that SharePoint was proving too successful. Customers’ needs were so great that they were driven to do things with SharePoint that the product simply wasn’t built to do. That isn’t to say SharePoint couldn’t do it—just that SharePoint 2010 wasn’t designed with the myriad insane use cases that flourished in the first year of the product’s release. Microsoft’s field support, consulting services, and product team were forced to respond to these use cases. Microsoft is unlike some other tech companies, in that it doesn’t just keep products in perpetual beta. Instead, it releases products thoughtfully, states its support policies clearly, and stands by them globally.

Microsoft’s teams did yeoman’s duty—in the midst of developing the next version of SharePoint—and responded to the cries from customers and ISVs by conducting exhaustive testing, and apparently building consensus across the SharePoint, SQL, consulting, and support teams to significantly revise its support for large content scenarios, and to build into Service Pack 1 important capabilities to support the scaling of SharePoint.

In short, the following supported limits were announced:
• Content databases of 200GB or less continue to be supported fully with out-of-box tools and functionality of SharePoint
• Content databases hosting document archives are supported with no limit to the size, other than that imposed by SQL Server itself
• Content databases hosting all other scenarios—loosely, “collaboration” scenarios—are supported up to 4TB
• A content database is supported up to 60 million items, including documents and list items. The supported limit for items in a single list or library (30 million) is unchanged.
• Size limits described above apply to the total size of data and content whether BLOBs are externalized using Remote BLOB Store (RBS) or not.

Now there is significant fine print associated with these changes. The bottom line of the fine print is that performance and architecture matter! Microsoft now makes no bones about the fact that out-of-box backup and restore tools, disaster recovery, and high availability options need to be considered, perhaps (or probably) supplemented, and very carefully architected if you want to support large content scenarios.

Where are we now? Now, SharePoint’s scalability limits—at least in terms of content—are very high. When you look at the performance guidance of .25 – 2.0 IOPS per GB of data stored in a content database, that’s faster hardware than many environments will have. We no longer have Microsoft “holding us back” from a support perspective. The team has put the stake in the ground far ahead of us, and has warned us clearly that we need to be very thoughtful about how we work to scale SharePoint. In the meantime, Microsoft is learning its own lessons about scalability as it implements Office 365, so I’m sure the next version of SharePoint will offer even better support for large content scenarios and for more insane use cases.

I think we—the community—owe the Microsoft support team a huge “thank you” for their efforts here. They get bashed a lot—and once in a while it’s deserved—so when they do something right, we need to cheer about it. In my opinion, they were thoughtful, disciplined, and most importantly responsive to what we are doing to their year-old product. Sure, we all knew SharePoint could do it, but it’s great to know that we now can, and be supported, without having to wait for vNext.

You can read the fine print, and all the details of the support changes in these resources:
• The announcement on the SharePoint Team blog: Data Storage Changes for SharePoint 2010
• The details on TechNet: SharePoint Server 2010 capacity management: Software boundaries and limits
• Bill Baer’s white paper: Managing Multi-Terabyte Content Databases with Microsoft® SharePoint® 2010

As you read these and other resources, don’t get too caught up in the issues related to BLOB externalization and Remote BLOB Store (RBS) quite yet. Obviously, in large content scenarios, a consideration of BLOB externalization is warranted. There is, unfortunately, still a lot of “noise” about BLOB externalization in the community, and next week we’ll tackle that debate and attempt to frame a discussion and real-world context so that you can make an informed decision about whether BLOB externalization is right for you.