Moving your Public Folders to SharePoint

Executive Summary: The decision to move your Microsoft Exchange Server public folders to SharePoint is not an easy one. Migrating is a complex labor-intensive undertaking that if handled incorrectly can result in significant business interruption. The guidance provided here can help you move your crucial data from public folders into SharePoint with as few problems as possible.

During the past year, I've worked with several clients that plan to use SharePoint as a replacement for Exchange public folders (and file shares, though not the topic of this article). Given the problems that organizations face with the uncontrollable growth of unstructured data, compliance requirements, and impacts on storage and operations, it’s a painful topic with no "magic bullet" solution.

Is there a rush to replace public folders? Probably not at the moment unless there are organizational factors surfacing that are forcing the change. For the time being, there's no time pressure for organizations to do so, because public folders will continue to exist and be supported until the end of the Microsoft Exchange Server 2007 product life cycle in 2016 or 2017. But there are compelling reasons for adopting a SharePoint-based solution: better presentation, search, and mobile access to name a few. But expectations must be managed carefully because migrating is a complex labor-intensive undertaking that if handled incorrectly can result in significant business interruption. The guidance I offer in this article is a general overview of the process that will help you plan, design, and carryout your migration with as few problems and pitfalls as possible.

The SharePoint Decision
SharePoint is not the solution for every enterprise. Organizations that are considering the move to SharePoint need to consider the following concerns prior to the move:

• Moving data from public folders to SharePoint is labor intensive.
• SharePoint stores the files in SQL Server so there are scalability concerns. Some companies have several terabytes of public folder data and file shares. You can find help in the TechNet article "Plan enterprise content storage"
• How do you tag the files so you can easily search for data? You can find guidance in the blog post "Searching Custom Column Values in MOSS 2007."
• Moving to SharePoint can be expensive. File servers are cheaper than SharePoint and SQL Server farms.
• You'll need additional tools, such as enterprise records management and archival
• SharePoint doesn’t support replication so you'll need a third-party replication product. Both Syntergy and Infonic   offer solutions.

Approach
Generally, I recommend that companies take a new approach to thinking about data--its classification, storage and retrieval. Organizations must think about information using a fine-grained approach as opposed to the "big bucket" approach of public folders and file shares. You need to consider how the information in your public folders map to your organizations Information architecture. Will you require sites for sales teams or project teams? Once data is migrated, how will you make sure the metadata is entered for each SharePoint Content Type? For example, information contained within public folders might consist of client information, product information, job related information, client information, or corporate. So how do you make sure that information is transferred to the SharePoint site?

Before conducting a migration to SharePoint, you need to make sure that the public folder (and file share) migration is handled according to organizational policy to prevent compliance issues and that you have a well-thought-out destination for the public folder data to prevent disorganization and users not being able to find the data they're looking for. For example, you need to set what data is permitted on public folders. (e.g., Microsoft Office documents but not MP3s), and what’s the policy for removal of violations? You also need to consider how the information will map between public folders and SharePoint Sites and Pages. Figure 1 provides a high-level view of how information (by functionality) maps to SharePoint.

Note that Figure 1 doesn’t depict how information maps specific to taxonomy (detailed Information architecture). In SharePoint, information is displayed in a more categorized and visible manner than public folders. For example, Contacts are placed in a Sites Contacts Web Part; documents are placed in a document library. Tools that migrate data from public folders to SharePoint will help with this classification and the creation of sites, but ensuring information relevance requires a lot of human involvement. Understanding this, your information architecture must address farms, Shared Service providers (SSPs), sites, pages, Web Parts (applications), content types, metadata, labeling, security, and content organization.

Before I dive into the methodology, I assume that you've already completed a SharePoint design that includes a detailed information architecture, system architecture, and operations plan. These steps are crucial for decision support and having a usable and compliant SharePoint navigation and search experience when you’re finished. If you need help planning your SharePoint design, the following resources are available:

• Information Architecture: "Logical architecture components" and "Information architecture in Office SharePoint Server."
• System Architecture: "Planning and architecture for Office SharePoint Server 2007," "Planning for Capacity Boundaries, Estimating Performance & Capacity Requirements... Additional Factors, and Tools" and various HP whitepapers
• Operations and Governance: Governance Resource Center for SharePoint Server 2007

The following sections outline a basic migration methodology, including the steps for each phase and hints for dealing with your project, because this project will change the way you handle your organizations data. Although it might seem like an overwhelming task, keep in mind that these steps are for a very large organization with gigabytes and perhaps terabytes of public folder data. You can always modify these guidelines to better suit your own organization, if necessary.

Phase 1: Project Initiation
In this phase, your goal is to build your team and prepare project documentation that will help steer your project effectively.

Establish a governance team. You need to establish a team and decision framework that will help you steer the rough waters ahead. Given that the project will touch just about every business unit, you'll require senior management to help facilitate the project's momentum and to address escalations and key decisions such as scope, resource scheduling, data retirement, prioritization, and business unit buy-in to name a few. The governance team should consist of executives from IT and business units, IT architects, the project management office, and purchasing, in order to be effective. A framework for escalations and making decisions must exist to facilitate the success of your governance program.

Develop a project charter. As with any project that deals with information, scope creep is your enemy because it could increase complexity and result in lengthy project schedules. Agreeing upon scope and priority is difficult especially if you have requirements that cover migration, security, and data cleanup. No one person can tackle public folder migration on their own--it's too complex. Assemble your team (Tiger team) with leads from the Help desk, data center operations, file servers, Storage, Exchange, SharePoint, records management, desktops, security and communications as a starting point. Public folder migration is far reaching and thus will impact the business. Your communication plan must address who, what, and when, and the business units that are to be migrated should be provided with communication early in the process. Expect push back due to concerns about business interruptions.

Phase 2: Requirements
In the requirements phase, your task is to develop a document and supporting material that will define the specific requirements of your organization and lead your team through the process of developing a site design. For example, your requirements might consist of migrating public folder data to SharePoint sites, establishing security guidelines and configuring SharePoint security accordingly, and ensuring compliance with information management policies and information architecture. The requirements document should contain the requirements for the team, tools, methodology and risk plan. Here are some things to remember when developing requirements:

Project Management Office (PMO). The PMO will have some insight to the project's initial scope, deliverables and time lines.

Compliance department. The compliance department might be actively involved in the cause of past audits that have exposed compliance issues, but if not, I suggest you meet with them to develop a list of compliance requirements. This could be a simple list of principles that must be incorporated into a design (e.g., being able to identify documents that reside on public folders that have legal impact, such as contracts).

IT department. You need to consider the IT department requirements for infrastructure and operations. For example, IT will be interested in capacity requirements and operational impacts (Help desk, monitoring, and n-level support). Note that it's important to involve the Help desk people since they'll need to help with support calls once the data migration begins.

Business requirements. Though often avoided by IT departments, working closely with the business early in the process is critical to managing perceptions and determining their specific needs. In many organizations the business has the final say. Winning them over through building rapport and trust is a must even if there is a history. This will help reduce the chance of push back or the project stalling later on during the actual migration. Also, don’t use the business as your testing ground. Use a lab and build a mockup of a business unit for testing.

Quality Assurance (QA). Many organizations have a QA process that can add significant time to your document acceptance process. Don’t forget to factor this into your timeline and documentation plan. Meet with the QA people to understand what you must provide them and when.

Inventory. The first major task for you will be to develop an itemized inventory of the public folders. To be successful, you’ll require a toolset. The toolset must be able to crawl and inventory the public folders and provide robust and customizable reporting. For example, being able to report on the amount of content, content types, content age, and orphaned data will be an asset. The tools you chose must be installed early on in the project so that inventory of the public folders can begin. Several vendors make tools for migrating public folders to SharePoint including Quest Software, Metalogix, and Tzunami.

Analysis. Generally your analysis will focus on the following:

• What data do you have?
• How much do data do you have?
• Where are the security and compliance risks?
• What data can be deleted to reduce storage consumption and operations costs?
• What data can be reused and placed within SharePoint?

Other projects. Most organizations have several projects underway at the same time. You must plan for this because collisions will occur and dependencies must be understood. Note that a meeting with the PMO will help you understand what projects are planned or underway and who you should begin discussions with.

Phase 3: Design
During the design phase your task is to develop a document and supporting material that will define the specific elements of your design. The design document will contain (depending on your organization's methodology) the design for the approach, methodology and support materials for your migration. This document should have tight linkages to your project requirements document and must address how the requirements listed in the requirements document will be addressed. The document should include the following items:

Tools infrastructure. Hopefully the toolset you use to inventory the public folders also has the ability to migrate the contents to SharePoint sites and log the results. If not, you must assess toolsets based on the requirements document. Your design must incorporate the technical infrastructure required to support the toolsets for the duration of the project. For example, how many servers do you require for the toolset? Do you need a workstation to act as the operator console? Does the toolset require a database? How much storage does the database require? Are agents required on the servers? What are the impacts to network bandwidth? In the case of low bandwidth networks, is an alternate means required? For example, one organization's network between the United States and Australia didn’t support the bandwidth required to migrate data. To address this issue, the server's drives were removed and shipped to the United States and the migration was performed there. If time permitted, perhaps a replication approach might have been a better approach.

Application remediation. Your design must incorporate tools and processes for dealing with applications that rely on the public folder infrastructure. Generally, the tools should be able to spot such applications; otherwise you must have a manual inventory process for collecting such information from the business units. There must also be a central record (e.g., a spreadsheet or database application) of applications to be remediated. Last but not least, your design should include a design for a development and QA environment for recoding and testing rewritten applications.

Public folder to SharePoint mapping. How will public folder data map to your SharePoint environment? Create a form in Excel that lists the mapping of the folders to SharePoint and any specific notes such as exclusions. Note that migration tools will attempt to create sites based on the public folder hierarchy and populate those sites with the content types contained within them. Expect to do some cleanup once the migration process is completed.

Work package. A work package contains a summary of the work assignment and any forms or checklists the user will require while performing the work.

Staffing. This section should describe the staffing model required to execute the public folder migration. Also, skill sets and experience must be listed here.

Training. The document should include training requirements for Help desk staff, IT staff who will support SharePoint, and SharePoint end users.

Testing. A well-defined test plan with scenarios and “How to…” checklists are a must to ensure that the migration occurred as planned. For example, you should perform the following tests:

• Data migration--Determine whether the public folder data was migrated and the target SharePoint site is in place.
• Security--Make sure the desired security model is in place.
• Data policy--Ensure that the data that does not conform to policy hasn't been migrated.
• Search/browse--Make sure that the data can be browsed and or searched using SharePoint.

Risk Management. A common way to address risk is to develop a risk plan document that lists each of the risks. An approach that works well is for you and your team to identify the risks, then analyze and rate each according to probability (high, medium or low) and impact (high, medium or low).

When developing your design document, lean on your Tiger team on a regular basis for design advice, reviews and sanity checks. Ultimately you want to have a weekly discussion with them so that you can stay informed about each others' projects, tasks, and roadblocks. You also want them to be co-authors for the document so they have some skin in the game.

Phase 4: Application Remediation
 This phase deals with the remediation of the applications discovered through the inventory process conducted with the business units during the previous phase. Generally, this is the most time-consuming process; each application is assessed to determine its specific requirements such as technology and level of effort. Generally, applications can be categorized as low, medium or high complexity according to the following guidelines:

• Low-- Your toolset and process has identified a simple solution. For example, simple code changes that remove specific public folder-related APIs and replace them with SharePoint-related APIs.
• Medium--Your toolset and process has identified a solution that involves moderate re-coding and testing. For example, simple code changes that removes specific public folder- and third-party product or Line of Business (LOB) applications-related APIs and replaces them with SharePoint-related APIs.
• High--Your toolset and process cannot identify a solution and therefore more detailed assessment is required. For example, the application requires recoding and additional products such as Office InfoPath to provide forms and SQL Server for a data repository.

I highly recommend that you perform application remediation early in the project. In large organizations, this phase should occur perhaps six to eight months in advance to the data migration phase.

Phase 5: Pre-Migration
This phase is all about making sure you (and your users) are ready to undertake the public folder migration. This phase is mostly about managing quality and risk. Also note that the premigration detailed steps are also specific to the toolset you choose. (For example, reporting is automated or a manual process). The following items must be double checked before migrating:

1. With your design complete its time to communicate your plan to IT and the business.
2. This is a good time to update your requirements and design documentation to reflect the realities of your organization. 3. Add resources or make project team changes based on how well people are working together, workload, and scope.
3. Schedule migration jobs to run as predefined times (off hours). Use caution to schedule jobs outside of other resource intensive jobs such as backup, virus scanning and indexing. Tools such as Quest Migrator offer flexible job scheduling.
4. It’s recommended that IT have an onsite presence to provide support, especially for complex requirements and or high visibility business users. It’s surprising how many companies forgo this.
5. Create a master record of applications to be remediated. This list is co-developed between IT and the business. This will include re-writes and applications to be replaced by a consumer off-the-shelf (COTS) product. The list must contain contact information and some sort of complexity rating.
6. The IT staff that will support SharePoint must be trained in SharePoint installation, administration, and so forth, and the business units must also be trained in general how-to and company usage policies prior to the actual migration.

Phase 6: Migration
This phase consists of the steps in the actual migration of public folder data to SharePoint sites. The actual steps for conducting the migration will depend on the migration toolset you've chosen because screens and options will be different. Therefore, I'll just generalize the basic steps, as follows:

1. Notify IT that the migration is about to occur, and notify the business unit that their data is about to be migrated.
2. Establish onsite presence and ready the Help desk.
3. Using the migration job schedule you created during the design phase, create the migration jobs and schedule them to run accordingly. To support your test plan, make sure you enable logging so that when the migration is complete, you can check for errors and deal with them. Also, use caution when configuring the security aspect of the toolset; going with minimum permissions is probably the best approach. Finally, set filters to prevent the migration of data that doesn’t conform to your organization's policy.
4. As the jobs run, monitor the jobs and the performance of the servers, storage, and the network. As the public folder data is migrated, it will tax these systems significantly unless the toolset provides throttling settings.
5. Here you execute the test plan you created during the design phase.
6. During and after migration, communication must be rigorously maintained between the migration team and the Help desk. Debriefing with the Help desk after migrations are complete for a business unit will help you learn and refine your methodology. Review the Help desk incidents to learn where improvements could be made. Also, expect some cleanup work to be done to fine tune the organization of sites and data. Depending on your organization's information architecture and expectations, this could be a lengthy process. Note that when escalation is required, you will require a process and clear ownership of tasks. From a governance perspective, you'll need a process for engaging with management in case you require their guidance or authority to obtain a decision or facilitate and action. An excellent book to help you plan your governance program is ‘IT Governance: How Top Performers Manage IT Decision Rights for Superior Results’ ISBN 1591392535.

Phase 7: Post Migration
During the post-migration phase, the organization is charged with maintaining the information architecture and enforcing the information management policy. Here are steps your organization can take to facilitate the success of these tasks:

1. Establish monitoring and reporting process and tools to ensure data quality, information architecture compliance, and security compliance. Most tools have predefined reports that will help you with reporting; the time consuming aspect of this task is reviewing reports and escalating issues to management.
2. Assuming SharePoint is new to your organization, you will have to ramp up staff and outfit your IT infrastructure with backup, monitoring, virus scanning, and other tools. And don’t forget about the SQL Server team--farm databases require regular maintenance to maintain performance. Refer to "Database Maintenance for Microsoft SharePoint Products and Technologies," which provides valuable information for SQL maintenance specific to SharePoint. 
3. Educate IT about the changes in technology and how they affect the services they provide. Impacts to SLAs and operations must be communicated and understood.
4. Educate staff and management about the changes in application technology and how they affect their jobs and their responsibilities. Involve human resources to make sure that both IT and staff take the educational training. Also, to facilitate user adoption, usage metrics should be added to the job descriptions of users so that usage in compliance with company policy can be measured. For example, project managers are responsible for uploading project related artifacts such as charters, schedules and design documents.

Making the Leap
Microsoft’s investment in public folders has noticeably declined in recent years, and SharePoint is clearly being positioned by Microsoft as the replacement platform. SharePoint offers many comparable features in addition to providing a platform for building, deploying, and managing applications. The decision to migrate your public folders to a platform such as SharePoint is specific to the organization's information strategy and is dependent on factors, such as the complexity of the current deployment, in addition to the availability of the necessary funds and resources.

Related Reading:

• Strategies for Migrating Public Folders to SharePoint
• SharePoint and Public Folders, Part 1
• SharePoint and Public Folders, Part 2