Parasoft SOAtest 5.0
A (Mostly) Fantastic Tool for Testing Web Services
By Matt Dinovo
One aspect of testing that is frequently overlooked is the testing of Web services. Lots of folks test Web services in the context of business operations, but not many test the raw service itself. This should be a critical part of performance testing any service oriented-based application (SOA), as it s vitally important to know how the service fa ade reacts under malformed messages, intense load, and other conditions.
I have employed various tools even going so far as to roll my own to fire messages at service implementations both in a functional and load scenario. When I was presented with the opportunity to review SOAtest 5.0 from Parasoft, I was eager to find out if, finally, a tool exists that was flexible enough to handle the multiple testing scenarios I mentioned above. The answer is ... almost.
Before I get to that, let me describe some of the features of SOAtest 5.0. SOAtest s primary function is to test Web services in a variety of ways: it can emulate the client or the server (by stubbing out services); validate WSDL documents; act as a unit, functional, or load test agent; and more all in an easy to understand user interface. Additionally, SOAtest has rich test-case management capabilities. Individual tests can be grouped into test suites that can be executed in sequence, allowing for various scenarios to be validated. Each test can have separate methods for determining input parameters and validating responses. When a test is executed, either by itself or as part of a test suite, any failures are clearly identified in the lower pane for analysis and remediation (see Figure 1).
Figure 1: The SOAtest test execution interface.
SOAtest is very strong in interpreting WSDL documents and provides several mechanisms to assemble requests based on WSDL. Once a WSDL is interpreted by the tool, you can provide input to the discovered Web services by using raw XML, either entered via the UI or from a file, or by using a dynamically generated user interface to interactively enter the input parameters for the request (see Figure 2). In addition, SOAtest supports data bound parameters in a variety of formats including a built-in spreadsheet paradigm to allow for pseudo-randomly generated inputs at execution time.
Figure 2: The dynamically generated parameter input screen.
One of the greatest capabilities of SOAtest with regards to test suites is the ability to extract values from service responses via XPATH, then use the result as inputs in subsequent service calls. In this way, a typical session from a client making several service calls in a row can be elegantly represented in the tool. Test suites can contain many different types of tests; however, for the .NET developer, SOAP client tests will be the most used test type. Responses can be validated, as well, and SOAtest provides validation wizards to validate individual parameters (via static comparisons or regular expression matching), the entire message, or the structure of the response.
Above functional testing, all tests can be executed in a load testing scenario. This functions in much the same way that load tests are executed in Visual Studio Team Test. You select the tests and/or scenarios you want to execute as part of the load test and then select the performance counters you want to capture as part of the test execution. During execution, SOAtest will capture the request/response metrics from the load test, as well as any performance metrics attached to the execution, and present the results in a unified view. SOAtest can simulate 100 virtual users on a single client.
WCF and SOAtest
As part of my review, I created several different services: traditional ASMX Web services and WCF services using basicHttpBinding and wsHttpBinding. I expected ASMX Web services to work flawlessly, and that s exactly what I experienced. When I flipped over to WCF services, the experience changed somewhat. Basic HTTP binding (SOAP 1.2) worked as expected, but services exposed through wsHttpBinding did not. I tried several permutations to attempt to get SOAtest to successfully send requests to WCF services exposed via wsHttpBinding, but to no avail. SOAtest was able to interpret the WSDL correctly and present me with an interface to set parameters for the request, but the actual execution of the request failed. wsHttpBinding supports more WS* standards than SOAtest, and I believe this is where the incompatibility lies. While this was disappointing, it was only a minor inconvenience I could still test WCF services using basicHttpBinding and a suitable workaround was to use multiple endpoints or to switch the binding when you deploy.
SOAtest is an impressive tool for testing Web services in a variety of capacities. The product boasts an impressive list of capabilities. However, because it is a Java-based application, I couldn t test certain features (but those were few). Over and above the everyday testing scenarios, SOAtest also allows for the establishment of policies that act as a kind of FxCop for WSDLs, penetration testing for security considerations, and even QoS and health reporting. Parasoft has committed to increasing SOAtest s .NET support, so I hope the issues of incompatibility with some WCF HTTP bindings will be resolved.
Having done performance testing on SOA applications, I truly appreciate the difficulty in being able to test only the service interfaces to see how they impact performance. SOAtest makes this task very easy as long as you keep things simple from a standards perspective (at least for .NET developers). If you are using ASMX or basic WCF services, you owe it to yourself to give SOAtest a try to see how it can help increase the quality of your system. You can try out SOAtest by downloading an evaluation version (88MB) from the Parasoft Web site.
Matt Dinovo is a Senior Solution Developer at Avanade Inc., a Seattle-based integrator for Microsoft technology that is a joint venture between Accenture Ltd. and Microsoft. You can reach him on his blog at mailto:mattdinovo.spaces.live.com.
Price: Enterprise Edition, desktop licenses, and training for a team of five costs around US$50,000.