The rise of public cloud computing and its adoption by enterprises of all sizes is presenting challenges to professionals who are charged with the security of the organization's data. One major issue is that individual departments and even employees can purchase public cloud services -- often by using a corporate credit card -- without the knowledge or oversight of the IT department. Such purchases can lead to significant governance challenges, introduce unknown risks, and even prevent the organization from meeting its statutory and regulatory compliance obligations.
Public cloud computing is desirable for many reasons, including increased IT agility, reduced time to roll out a new product or service, access to the latest technology not available inside the enterprise -- and even a strategy to work around restrictions put in place by the IT departments, such as a limit to the size of email attachments or the types of files that can be sent or received through the email system. For these reasons, many IT departments are considering deploying private clouds, which departments can access and use instead of public clouds. Examples on record include State Street Bank (which expects to see significant savings as well as improve operational efficiency and security of customer data), engineering and construction firm Bechtel Corporation, and chemical company Sinochem Group.
However, private clouds aren’t inherently more secure than public clouds and can even be far less secure. In this article, I'll discuss some pitfalls and make recommendations for securing private clouds.
Overview of Private Clouds
One difficulty that security professionals encounter is a variety of perceptions amongst IT staff, senior management, and end users about what a private cloud really is. For example, many believe that private clouds are exclusively on-premises, residing in a data center that the organization controls, and therefore are more secure than a public cloud, which is hosted by a public cloud provider. Another common misconception is that a private cloud always uses virtualization to create a pool of virtual machines (VMs), which can be allocated to departments and users as needed.
Related: Top 10 FAQs for the Private Cloud
Although it’s fair to say that most private clouds that are deployed or under consideration today are on-premises, provide Infrastructure as a Service (IaaS), and use virtualization technology to create a pool of resources that can be allocated as required, the reality is that a private cloud is simply a cloud that is dedicated to an organization for its sole use. The private cloud can be on- or off-premises, and indeed several extremely reputable private cloud providers use their own data centers to host private clouds for their customers. Some of these private cloud providers (e.g., Microsoft) also offer public cloud services. Nor are private clouds limited to IaaS offerings. Many can and do offer Platform as a Service (PaaS) or Software as a Service (SaaS) as well. Virtualization, especially in SaaS clouds, is not a prerequisite. Organizations might find it useful to consult the National Institute of Standards and Technology's (NIST's) Special Publication 800-145 for a better understanding of cloud computing, including private clouds. (See the sidebar "Cloud Definitions" for more information.)
Threats in Private Clouds
The Cloud Security Alliance (CSA) published its Top Threats to Cloud Computing research report in March 2010. Although the document is several years old and is currently being revised, it remains extremely relevant today. Of the seven threats identified in the report, each applies to private clouds, whether hosted on- or off-premises. All seven apply to IaaS, six apply to PaaS, and five apply to SaaS clouds. The threats, in no particular order, are as follows:
- Abuse and nefarious use of cloud computing
- Insecure APIs
- Malicious insiders
- Shared technology vulnerabilities
- Data loss or leakage
- Account, service, and traffic hijacking
- Unknown risk profile
The CSA has published guidance and tools that cloud consumers and providers can use to jointly tackle these threats. These tools are freely available on the CSA's website.
The reported threats aren’t considered the only threats to cloud computing, though. Experience with private clouds has highlighted specific areas of concern for enterprises. As I mentioned earlier, most private clouds provide IaaS and use virtualization to make VMs available to departments and users within the organization, on an as-needed basis. Both Microsoft and VMware provide the necessary technology to build private clouds from the ground up. Excellent open-source tools, such as OpenStack, work with a range of hypervisors such as Citrix Xen and Linux Kernel-based Virtual Machine (KVM), as well as Microsoft and VMware hypervisor technologies. For that reason, I will focus on IaaS risks for the remainder of this article.
Risk #1: Abandoned VMs
Private clouds have led to an explosion in the number of VMs in existence. Private clouds are used to develop and test line of business (LOB), or Tier-1, applications and customer-facing web-based applications, as well as to host production environments. Organizations often develop entire libraries of VMs that can be deployed at a moment's notice, to handle additional workloads or to accommodate specialized testing. Creating new VMs can, in some cases, be as easy as copying the configuration files that define a VM and the file or files that comprise its virtual hard disk (VHD). This explosion has led to a problem whereby VMs are created and used but rarely deleted. When a VM is no longer necessary, it is often simply turned off and left intact, just in case it ever needs to be used again. This approach might mean placing it back in a library. This phenomenon is facilitated by the relatively low cost of storage used to house the VM libraries.
The risks to the enterprise from such practices are many. For example, VMs that are simply turned off typically are not turned back on to apply routine software updates. When a VM is used only during periods of peak demand, it might go weeks or even months between uses, by which time it might have several critical vulnerabilities for attackers to exploit as soon as it comes online.
Every organization should put in place a set of procedures to ensure that VMs have all software updates applied before being used in production. Unfortunately, many organizations do not create a production-like environment in which the VM can be brought online, updates can be applied, and then regression can be tested before the VM is moved into production. Either these organizations haven’t planned for such an environment, or they don’t have the time or resources to create one. When in production, each running VM should be treated just like a physical server and should have all appropriate software updates, maintenance releases, and so on applied, to ensure that no vulnerabilities can be exploited by attackers.
Risk #2: Exposing Sensitive Data
Another common problem with the explosion of VMs in private clouds is the amount of highly sensitive data that is stored on these VMs. This sensitive data can include personally identifiable information (PII) such as usernames, passwords, Social Security numbers, or credit card details. It can just as easily hold protected health information (PHI) or the organization's business secrets, including source code and product plans. When a VM is running in a production environment, access to a live VHD is tightly controlled by the hypervisor or other virtualization software or hardware. The VHD file (or files) is often locked for exclusive access by the hypervisor, to prevent corruption. The OS in the running VM implements logical access control, preventing access to sensitive data just as an OS would do on a physical server when users attempt to access the server over the network or to interactively use a remote command-line or desktop interface.
Plenty of tools can be used to mount a VHD as a drive letter or mount point when not in use. These tools can allow an attacker unrestricted access to the data on the VHD, including extremely sensitive data. However, malicious attackers aren’t the only ones accessing offline VHDs, looking for sensitive information that organizations need to be worried about. Private clouds and virtualization make it easy for developers to get access to production servers that are, in actuality, virtualized servers, for the purposes of testing and debugging simply by copying the configuration and VHD files to their development environment. There are many restrictions in place about access to various types of data, and the organization might need to disclose, to regulators and even to customers, the inadvertent access to data by developers. This disclosure can bring the prospect of fines or other penalties, and the loss of customer goodwill and business. For these reasons, access to the configuration and VHD files that make up VMs should be restricted and monitored. Although it comes with a performance overhead, sensitive data should be encrypted when stored in a VM, and the encryption keys should be stored elsewhere, such as a network-based encryption device (e.g., a Hardware Security Module -- HSM). When it is impossible to encrypt data stored within a VM (e.g., when running a commercial off-the-shelf -- COTS -- application that doesn't support encryption in a VM), it might be possible to encrypt the VMs' configuration and VHD files, depending on the virtualization technology used. Last, logical access controls such as discretionary ACLs (DACLs) should be used in the host environment and in the library, to restrict access to the VM files when not in use.
Risk #3: No Network Access Controls
A third problem commonly encountered in private clouds, especially in private clouds where VMs can be moved from host to host to allow for maintenance of host machines or to ensure uptime in the face of host failure, is that the networks in which the machines run are relatively flat, perhaps using RFC 1918 Class A addresses in the 10.x.x.x or Class B addresses in the 172.16.x.x – 172.31.x.x ranges. Flat networks make for more-simple-to-build and easier-to-manage physical networks connecting the VMs. These networks also help to ensure connectivity from one VM to another, to a physical machine, or to the Internet, regardless of the physical host on which the VM is running.
Unfortunately, flat networks rarely have much, if any, security in place. There are fewer routers or firewalls on which to place ACLs, and there’s no single point at which Intrusion Detection Systems (IDSs) or Intrusion Prevention Systems (IPSs) can be deployed to monitor all network traffic. The lack of ACLs and IDS or IPS means that as soon as an attacker or piece of malware is in the network, it can move around with relative ease and with little chance of detection.
Although network-level controls are preferred and even a best practice as part of a defense-in-depth strategy, the lack of security in the network can be compensated somewhat through the use of host-level firewalls within the VMs. Unfortunately, configuring these firewalls can be a management nightmare. There is a lack of tools for the task, and many VMs reside offline in a library, and so cannot be easily managed. Some virtualization software does allow you to configure network access to VMs and comes with management tools to support configuration and maintenance. And several industry alliances promote best practices in network security through reference architectures and tools for networking components and virtualization software from specific vendors. However, these are proprietary and won't work in moderate to heavily heterogeneous host environments.
Risk #4: Lack of Anti-Malware
Another security problem that is routinely encountered in private clouds is a lack of anti-malware defenses. There is much confusion and misunderstanding about the role of anti-malware software in IaaS clouds -- a holdover from when virtualization first became mainstream. For example, many organizations won’t employ it at all on the host or in guest VMs, believing that it’s either not required, will cause performance issues, or might even cause a VM to fail.
The truth is that anti-malware defenses are required and need to be carefully deployed to ensure smooth operation and protection of host machines and guest VMs. Careless deployment of anti-malware software to a host machine can cause significant performance effects and can cause a VM to be flagged as malicious (either because it’s infected or because the malware signatures of the anti-malware software in the guest VM are detected and flagged as malware).
Anti-malware software on the host machine should be configured to exclude the scanning of files that comprise VHDs and the processes that are associated with the virtualization software. Anti-malware software in VMs needs to be configured so that it doesn’t cause performance problems on the host machine, which could affect performance across other guest VMs running on that host. This requirement might mean configuring the anti-malware software in VMs to scan for malware in files only during off-peak hours, and limiting the amount of CPU and memory resources that the anti-malware software uses, if possible.
Virtualization best practices, including dedicating a CPU core to each VM and placing each VM's configuration and VHD files on separate physical SCSI hard disk spindles, will largely mitigate the possibility that anti-malware software in a VM will have a performance impact on the host machine and other guest VMs. As with applying software updates, which I described earlier, the organization needs to ensure that offline VMs in a library are routinely brought online to obtain the latest anti-malware software and signatures. Otherwise, when brought online, the VMs might fail to detect newer malware.
Risk #5: Decentralization
The final security concern that I want to highlight is a lack of understanding about how departments and individuals use private cloud resources. When an organization deploys a private cloud, there is often a tendency to decentralize the IT department, assigning IT staff to business units and departments that might also hire their own IT staff, architects, and application developers. The role of the IT department descends into one whose primary purpose is simply to keep the private cloud infrastructure up and running and to allow the business units and departments to manage their own VMs. In such environments, business units can begin to make compliance- and risk-based decisions about what to put in the private cloud and which controls and protections to put into place. Unfortunately, these individual business units and departments, as well as whatever is left of the central IT department, are unaware of other business units' and departments' use of the shared private cloud. As a result, they might introduce an unacceptable level of risk to the entire organization.
For example, suppose that one business unit deploys a customer-facing application that collects and processes credit card information, taking care to ensure that the appropriate controls are in place. That effort can be imperiled by another business unit that deploys a customer-marketing website for a seasonal campaign but does not deploy suitable security controls for an environment that processes credit-card data. Or suppose that a business unit or department deploys an application that requires high security because it processes customer PII. However, VMs are stored in a library, with little to no protection, when not in use. To ensure that no business unit or department introduces unnecessary risk, a centralized function must understand which groups are deploying what into the private cloud, ensure that the private cloud infrastructure can support all security requirements, and ensure that no group is deploying insecure applications. To accommodate environments in which differing business units have differing security requirements, an organization can choose to either partition the private cloud into high-, medium-, and low-security segments, or can consider deploying more than one private cloud.
Balance the Benefits and Risks
Private clouds can bring many benefits, including reduced costs, increased business agility, the ability to empower individual business units and departments, and a closer alignment of IT and business needs. But private clouds can also introduce new risks. Organizations that are considering private clouds -- as well as those that have already deployed them -- should consult guidance such as that made available by the CSA and the software manufacturers whose products and tools the organizations are using to build out their private cloud environments. Other resources that might be of use are those made available by public cloud providers such as Google, Amazon, Microsoft, HP, Verizon, and Rackspace. These resources are intended primarily to address the security and privacy concerns of potential customers but also contain a wealth of detail about how these vendors built out their own cloud infrastructures and the types of controls that they put into place to secure customers' data.
