Among the more important business decisions that IT departments are called on to make is selecting the most effective application to solve a given business problem. For most organizations, Commercial Off-the-Shelf (COTS) applications are the option of choice. Unless the business problem is novel or highly specific, many applications likely exist to fulfill the business need. Which one should you choose? This is the kind of question that can make or break an IT executive's or manager's career. And thanks to the cost of purchasing, customizing, and deploying applications, you might have only one chance to answer it.
Level the Playing Field Selecting a new application, whether to replace an existing application or to be installed as a new deployment, is almost always a hotly contested project. The business groups that will rely on the application have as much riding on the decision as the IT department does. Both groups need to be closely involved in the decision-making process. A more effective solution than making ad hoc decisions is to evaluate all applications according to the same standards. Doing so enables business and technical decision makers to compare applications to one another on a more realistic basis. Evaluation standards are also essential for explaining to management why the application that you ultimately recommend is the best one for the job. Your standards should encompass, at a minimum, the following 10 categories.
Cost. Although the concept might be counterintuitive, cost is often the most difficult aspect of an application to evaluate because frequently, sticker price is only the tip of the cost iceberg. In addition to the upfront price of the application, per-seat licenses, upgrade costs, volume discounts, annual fees, expected support costs, customization, training, and deployment all increase the per-seat cost of the software. During your evaluation, be sure to quantify as many of these costs as possible to get an idea of the total per-seat cost of owning the application. Frequently, you'll find applications with the lowest upfront cost can incur much higher total cost of ownership (TCO) than applications with a higher price tag.
Business requirements. To determine how well an application meets core business requirements, have the groups that will be using the application create process-flow diagrams of their business activities and of how they need to interact with the application. Because it's easy to leave out steps when describing business processes, having a visual representation, such as a flowchart, enables members of the business group to quickly verify the accuracy of the workflow. Capture these common scenarios, then ask the business groups to identify their dream features for a new application, given their knowledge of the business. Map this information against the capabilities of the applications you evaluate.
Vendor commitment. You don't want to buy, design, or deploy an application that will be discontinued or radically changed in the near future, nor do you want to buy an application from a company that might not be in business in 3 years. It's important to know how committed the vendor is to the application. Other important information to know is whether the application has planned future versions or updates. If so, what's the schedule for those updates? Does the vendor have a process for discontinuing support should it no longer support the application in the future? What is the technical road map for the application? How financially stable is the vendor?
Architecture. Understanding the architecture of the applications you evaluate will better equip you to determine how each application will interoperate with your existing network and applications and better prepare you to locate potential security or reliability concerns. You need to know how the application works and how it integrates into your platform for security, reliability, and other services. Understand the architecture's design criteria for common scenarios, such as geographic distribution of users. Determine how the application scales up and out and whether it uses standard protocols. Finally, determine whether the application meets your organization's legal or regulatory compliance requirements.
Interoperability. An application's cost can greatly increase if it doesn't interoperate natively or easily with your existing or planned platform or other applications. Furthermore, applications that don't interoperate well are more likely to be adversely affected by changes in platform or application. Assess whether the applications you're evaluating work well on the platform that your organization is currently using and on any that you plan to use. Determine whether the applications natively interoperate with other applications that your organization uses and what, if any, middleware might be required for other forms of interoperability. Do the applications use commonly adopted standards or proprietary protocols (for communication, data storage, or management interfaces)? Proprietary protocols can make interoperability with other business systems challenging and potentially require middleware applications, which can increase a solution's cost.
Security. To ensure that your organization can properly protect the information that the application you select will use and store, you need answers to some crucial questions about the application and its vendor. For example, does the vendor use a security process that has specific measures to protect the applications against attacks? How do you secure the application, and what resources are available to help you do so? What mechanisms authenticate users and computers and authorize access to data? Do these mechanisms work with your platform? How is data secured both in place and when it's being transmitted over the wire? What auditing services does the application provide? How can you assess the security of your deployment? How does the vendor handle reported vulnerabilities?
Reliability. Because ensuring availability is often the single most important responsibility that the IT department will have in administering an application, be sure that you are comfortable both with the reliability of the application you select and with the reliability features the application provides. Understand up front which hardware and software reliability mechanisms, such as clustering and load balancing, you can use to ensure that the application is available according to business needs. Determine how the application alerts administrators when it fails and how it recovers from failure. Ask the vendor what types of reliability testing it has conducted on the application. Know how you can measure the application's uptime.
Usability. Applications that are difficult for users to work with will incur costs in productivity loss and training. In addition, you risk political fallout from the business group that employs the application if users reject it. Always create focus groups and usability tests with a subset of users when evaluating applications. Determine whether users can use the application easily and whether and how much user training will be required. Find out what user training the vendor or the vendor's partners provide. Know whether the interface can be customized or personalized and whether the application is available in all of the languages in which your users communicate.
Manageability. Applications that aren't easily manageable or aren't manageable according to the way your IT staff is resourced can cause TCO to rise or can result in lower then expected levels of security and reliability. When evaluating an application, determine whether your organization's administrators possess the skills to manage it. If not, is training available? Determine which management tools the application supports. Understand the options administrators can use to manage the application locally and remotely and which automation options the application supports. Know how management operations are audited and whether application management can be delegated to non-administrators. Finally, evaluate how the application reports its health.
Supportability. The ability to quickly and efficiently resolve support problems with the application you choose is crucial to ensuring its availability and controlling its cost. Ask vendors what free support options they provide for their applications, and make sure you distinguish these options from paid support options. Evaluate how useful a product's documentation is and determine whether active community support forums exist. Look for and evaluate native diagnostic software. Finally, understand how your organization would escalate a support concern with the application.
Compile the Scores To construct a value assessment for each application you evaluate, employ a consistent criterion. One effective method for doing so is to use a five-point Likert scale to rank an application's performance in each category relatively from worst to best. This method makes comparing applications easy and also gives you a basis for mathematical comparison. To use this method, grade each application in each category with a number between 1 and 5 according to the following scale:
Worst Bad Neither good nor bad (average) Good Best
After you've assessed all the applications you're interested in, total the scores for each. In general, you can safely eliminate any application with an average score of 3 or lower. Applications with scores this low aren't suitable for your organization. For applications with average ratings greater than 3, you can perform more in-depth analysis. For example, you can create a cost-per-rating point calculation by dividing an application's projected cost per seat by its average rating. The application with the lowest resulting number is the one that gives you most value for cost. For ease of comparison, it can be useful to also draw up a short list of pros and cons for each application within each category.
Come Up with a Winner By using a structured approach to evaluating applications for purchase, you'll be better able to compare applications meaningfully to arrive at more informed and objective decisions. This approach also builds a nice framework for presenting your research to executives and other key decision makers. Now, all you have left to do is deploy the application.
