And when we say software testing, we’re not talking about opening your website or app a few times on your development box to see if it seems to work, then calling it a day. Modern software testing requires automated solutions that can quickly and systematically evaluate how well an application or website performs across a variety of browsers, operating systems and hardware devices. It also means running tests again each time you are preparing an update for an application.
There are a number of automated testing solutions available to meet this need -- so many, in fact, that it might be hard to choose which one best fits your needs. That’s why ITPro Today has prepared this Software Test Automation Tools Buyer’s Guide.
What’s in a Test Automation Solution?
A software test automation solution is any type of platform, framework or application that automatically tests how an application behaves.
Software test automation tools are the opposite of manual testing tools, which involve firing up a website or application by hand and performing an evaluation manually to determine whether it behaves as required.
A number of open source test automation frameworks, such as Selenium, Appium and Cucumber, exist to help developers and quality assurance teams write automated tests that can analyze how an application behaves.
However, running automated tests requires more than just a framework to write the tests. You also need hosting environments in which to run the tests. That’s where commercial software test automation platforms come in. They are designed to make it easy to execute automated tests across a wide variety of environments.
Ideally, you should run tests in each of the operating system/browser combinations that your end users use, to verify that your application behaves as required under each configuration. Before you deliver your application to your users, it should work as well on Firefox running on Windows 10 as it does Chrome running on the latest version of macOS, for example.
And if your application runs on mobile devices, testing it across a range of device types is important, too. You can do this using an emulator, which creates a software-defined, virtual environment that mimics a given type of mobile device. Or, for the most accurate results, you can run tests on real mobile devices.
Variables in Software Test Automation Tools
All major test automation platforms offer a common set of core features. They provide hosting environments for running automated tests across a range of operating system, browser and hardware device combinations using a “device cloud.” A device cloud is a network of servers and (in most cases) mobile devices that developers can connect to to run automated tests. A device cloud saves developers from the hassle of having to set up their own on-premises testing infrastructure, which would be very expensive and complicated to maintain if theyneed to test apps on dozens of types of configurations.
Virtually every automated testing platform available today works with all the major open source automated testing frameworks. That’s why we didn’t include supported frameworks as part of our guide; there is not much variation in that category.
However, commercial test automation solutions vary in many other respects, including:
- Whether they support desktop application testing, mobile application testing or both (most modern platforms support both).
- Their support for testing on local devices. While all of the platforms we reviewed offer device clouds, only some allow developers to use their platforms in conjunction with local devices, too. Local-device testing is sometimes necessary for security reasons, or in the rare case that the vendor’s device cloud doesn’t contain the specific device you need to test on.
- Whether they allow headless testing, an automated testing technique that saves resources by running tests for browser-based apps without spinning up an actual browser. While headless testing is not the right solution for all types of tests, it can make testing faster and more efficient.
- Their support for testing on real devices, in addition to simulated or emulated devices.
- Support for load testing, which evaluates how well an application functions during times of heavy demand. Few of the platforms we evaluated offer load testing. There are many other vendors out there, such as BlazeMeter and LoadRunner, that specialize in load testing but don’t offer the other testing features we focused on in our guide.
- Support for running tests manually. Although automated testing is much faster and more scalable, there are still cases where developers might want to perform manual tests to track testing results very closely or test features (such as usability) that are difficult to evaluate in an automated fashion.
- Cost, which varies widely between platforms. Most vendors set prices based on the number of tests that users wish to execute at once, as well as the amount of time they spend executing tests. Most vendors offer multiple pricing plans; our guide includes basic pricing information for the plan that a small or medium-sized team of developers is most likely to use.
With that information out of the way, it’s time to download our free Software Test Automation Tools Buyer’s Guide (registration required).