Change control is more important than ever as an IT control to mitigate many different types of risks, including compliance, continuity, and security. Without effective change control, system upgrades and other changes can wreak havoc on an organization's IT infrastructure and impede the wheels of commerce. Change control, always considered a best practice, is now the law in some cases. For example, managers of publicly traded companies are obligated to report events that may seriously affect a firm's financial outlook. But these managers can't determine whether changes in IT fit those criteria unless a change control process identifies them as such.
Compliance and continuity, however, aren't the only reasons to implement strict change control processes. Change control also is crucial to maintaining the security your systems. I've repeatedly seen situations where system upgrades, changes in IT staff, and other events result in the discontinuation of crucial security controls. In this article, I'll discuss the major elements of a modern change control program and relate these elements to key components of the Windows environment. Change control includes impact analysis, testing, a review and approval process, and before and after reports to management. Offline from the core change control process you may also have detection and verification processes.
Impact analysis. In an impact analysis, you determine the scope of a change and identify all systems and processes that will be affected by the change. For instance, when a new patch is released for Windows you must determine which systems actually need the patch. If the patch is specific to, say, Active Directory replication, an impact analysis should conclude that domain controllers are the only systems that need to be updated but that any systems (and the processes supported by those systems) relying on domain authentication or DNS could be affected if the patch introduces a stability problem. Or, in another instance, you might find it necessary to move an accounting application off an existing server to a new dedicated server. A thorough impact analysis will identify every system or application that references that server (e.g., monitoring systems, other applications that upload or download transactions) so that they can be updated to reference the new server.
Testing. Seemingly simple changes to technology often have unintended consequences or require additional updates that an IT staffer never thinks of until something breaks - things missed by impact analysis or things due to an incomplete understanding of how a technology works. The only sure fire way to avoid breaking the production environment is to try breaking a closely duplicated test environment. Unfortunately, mirroring a complete corporate data center in a test environment isn't possible. But with some imagination, you can find ways to test the major aspects of a change. For instance, in Active Directory changes to group policy objects can have far reaching effects on the security settings of thousands of computers. If a test AD environment isn't available, you can easily set up a test branch on your organizational unit structure and create test user accounts, computers, and group policy objects to simulate group policy changes.
Review and approval. A change control process can look great on paper but unless the process is enforced through peer review and supervisory approval, it can easily become a routine "go through the motions" process with only token adherence. While impact analysis and testing are usually coordinated by the IT staffer assigned the project, someone else should handle the review and approval process for the sake of separation of duty. When a qualified colleague reviews your planned changes, many problems are averted for two reasons. First, when you know a peer is going to look at your work you are naturally more careful. Second, there's no substitute for having another pair of eyes check your work. But real enforcement of the change control process happens when both the implementer and reviewer are required to state on record to their supervisor that the proper impact analysis, testing, and review was performed before getting final approval to make the change.
Reports to management. Communication is important during an impact analysis and directly before a change. All interested parties need to be given an opportunity to report potential effects of a change that aren't known by the individual or team assigned to the project. Then, before the change is made in production, everyone needs to be on board and ready to perform related steps such as configuration changes or system restarts. You also need to report to management the potential risks and impact of a change before actually executing the change. Only with such reports can management comply with legislation such as Sarbanes-Oxley.
In addition to good communication, you also need good record keeping. Auditors and regulators don't consider processes to be in compliance without seeing supporting documentation. With a little imagination you can create a self-documenting change control process using a collaboration tool like Microsoft SharePoint Services. You can easily use SharePoint's list, discussion board, and calendar features to create a simple change control system. For example, you can create a list where each change or project is documented in a line item with appropriate fields for status, person responsible, reviewer approver, and so on. In a discussion board, the person who owns the project can initiate a discussion where he documents the change and his impact analysis. Interested parties who subscribe to the board can weigh in with additional considerations. The reviewer replies with his observations and then the approver documents his approval. By communicating via a discussion board instead of through emails, the process documents itself. The list becomes a structured index of all changes, with the corresponding discussions available for more detailed research.
Detection and verification. Detection and verification runs on a separate track from the core change control process. Monitoring applications, whether they are log-based, base-line scanners or SNMP monitors, provide an excellent final layer of defense against unauthorized changes that slipped through the change control process or - more importantly - malicious changes by an attacker who has compromised your network. While there may be some overlap between detection and verification, verification (sometimes called re-verification) is the process of periodically verifying the validity or integrity of your network's components. Examples include scanning system configurations to make sure they still comply with standard settings, group memberships for inappropriate members or dormant user accounts. Getting the budget for monitoring applications and scanners can be difficult, but as auditors and regulators push harder and management becomes more educated about IT security this area of change control will continue to grow.
Change control isn't the fun part of IT but there's no substitute for it. IT departments that follow these principles faithfully have a solid reputation in their user community for stability and therefore reap the benefits of trust. Management must exercise care that change control doesn't just become paperwork that nobody ever looks at. To be successful, change control must start with management's commitment and participation. Initially, change control might seem to slow down the rollout of new technology and adversely affect IT's ability to be responsive and flexible. But as a business grows so does the complexity of its IT infrastructure; your ability to remain flexible and responsive will depend on the maturity of your change control process and its ability to manage the increasing complexity.
