Testing Web Services

Web services are essentially remote method calls that use HTTP and SOAP protocols. They provide ASP.NET developers with an elegant new way to construct applications, and are being used progressively more often in enterprise environments. In conversations with colleagues, however, I discovered that the techniques to test Web services aren't well known. In this article I will show you how to quickly write a powerful test app that verifies a Web service's functionality. For more information on web services, see "Web Services Client Authentication " and " Web Service Essentials ."

This article assumes that you're familiar with creating ASP.NET Web applications, have a basic understanding of ASP.NET Web services, and have intermediate familiarity with the VB.NET language.

The best way to show you what we'll accomplish is with two screen shots. Figure 1 shows an ASP.NET Web application that calls two methods in a Web service to provide data to the user. Behind the scenes are a remote CountTitles method and a GetTitles method. Both methods query a remote SQL database table of book titles. The user enters a search term in the TextBox control. If the user clicks the Count Titles button, the application queries the table and returns the number of book titles that contain the search term. If the user clicks the Get Titles button, the application obtains a DataSet object containing all information about titles that contain the search term and then displays the title strings. In this example, searching for "computer" returns the five titles shown.

Figure 1: An ASP.NET Web application that uses a Web service.

Manually testing the Web service methods through the Web application would be extremely tedious, time consuming, and error prone. Instead, we can test the Web service by programmatically sending input to the remote methods and examining the responses. Figure 2 shows a console application that does just that. (All of the code shown in this article is available for download; see end of article for details.)

Figure 2: Automated testing of the Web service.

In the sections that follow, we'll walk through the underlying Web service methods to understand what we're testing, briefly examine the example Web application to understand why we need to test it, and carefully go over the console application to understand how to test Web services. I'll conclude with a discussion of some of the ways you can extend this technique and use it in a production environment.

The Web Service

One of the most common uses of Web services is to provide an interface to proprietary data. In this example I am using the sample database named "pubs" that comes with Microsoft SQL Server. It contains data for a hypothetical publishing company. Figure 3 shows the book data in the table named "titles" displayed using the Enterprise Manager tool.

Figure 3: The titles table in SQL Server's Enterprise Manager.

It's easy to imagine that a Web service that exposes data like this could be very useful in a variety of ways. I created a simple Web service with two methods. Figure 4 shows the code for a CountTitles method that returns the number of book titles that contain a target string.

 _

Public Function CountTitles( _

   ByVal target As String) As Integer

 

   Dim MyConn As SqlConnection = New SqlConnection( _

    "server=(local);database=pubs;user=testUsr;pwd=secret")

  MyConn.Open()

   Dim MyCommand As SqlCommand = _

     New SqlCommand( _

      "SELECT COUNT(*) FROM titles WHERE title LIKE '%" & _

      target & "%'", MyConn)

   Dim result As Integer

  result = MyCommand.ExecuteScalar()

   Return result

End Function

Figure 4: This Web service method, CountTitles, returns the number of book titles that contain a target string.

I used the VB.NET language, but any .NET language can be used when creating Web services. In this case the database is on the local machine, but in practice it would likely be located on a dedicated server (there are no differences as far as the techniques presented in this article are concerned). Beginners are often amazed that merely by using Visual Studio .NET and adding the attribute in front of a method you can create a remote method.

The GetTitles method in our Web service is more complex; it returns a DataSet consisting of all book information in the titles table in which the title string contains a search string. The code is shown in Figure 5.

 _

Public Function GetTitles( _

   ByVal target As String) As DataSet

 

   Dim MyConn As SqlConnection = New SqlConnection( _

    "server=(local);database=pubs;user=testUsr;pwd=secret")

   Dim MyAdapter As SqlDataAdapter = New SqlDataAdapter( _

    "SELECT * FROM titles WHERE title LIKE '%" & _

    target & "%'", MyConn)

   Dim MyDS As DataSet = New DataSet

  MyAdapter.Fill(MyDS)

   Return MyDS

End Function

Figure 5: GetTitles returns a DataSet consisting of all book information in the titles table in which the title string contains a search string.

If you are new to ADO.NET, you can think of a DataSet as an in-memory representation of the titles table. Conceptually, the DataSet object closely resembles the representation in Figure 3; each row represents the data fields for a book, and each column represents an individual field (title, price, etc.).

To test our Web service, we need to analyze the functionality of the CountTitles and GetTitles remote methods. Each accepts a single string as an input parameter. CountTitles returns an integer, so it's fairly easy to determine if we get the correct result, but GetTitles returns a DataSet object, so it will be slightly harder to deal with.

The Web Application

The .NET Framework makes it easy to call Web services from an ASP.NET Web application. To produce the application shown in Figure 1, after creating a new Web application project, I simply added a Web reference to the Web service I created. Behind the scenes, Visual Studio .NET generates all the code you need to connect to and call the methods in the Web service. After that, instantiating the Web service is just like instantiating any other object.

The code in Figure 6 shows how easy it is to invoke the GetTitles method of the Web service. I used the default name of WebService1; its namespace is localhost, because it's on the local machine. GetTitles returns a DataSet object, so the code iterates through each row and adds the value of the title column to a ListBox control. The code that invokes the CountTitles method is similar; after creating a WebService object, you can call its CountTitles method, capture the returned result, and display it.

Private Sub Button2_Click(ByVal sender As System.Object,

   ByVal e As System.EventArgs) Handles Button2.Click

  ' Get Titles.

   Dim MyWebService As localhost.Service1 =

     New localhost.Service1

   Dim MyDataSet As DataSet =

    MyWebService.GetTitles(TextBox1.Text)

  ListBox1.Items.Clear()

   For i As Integer = 0 To MyDataSet.Tables(

                            "Table").Rows.Count - 1

    ListBox1.Items.Add(

      MyDataSet.Tables("Table").Rows(i)("title"))

   Next

End Sub

Figure 6: It's easy to invoke the GetTitles method of the Web service.

It's possible to test Web service methods by using the Web application that calls the methods, but my experience shows that it's much better to test the Web service directly. The problem is that there are often interactions between the application and the underlying Web service, so when a test case fails it can be difficult to determine if the problem was with the Web service, the application, or the connection between them. In testing, this is called the validity of the test. Now this is not to say that you shouldn't test the Web application, but rather that you also need to test a Web service independently of the application that calls the service.

The Test Automation

In the previous sections we saw that a Web service is essentially a collection of remote methods, and that implementing them and calling them in a .NET environment is similar to writing and invoking ordinary local methods. I contend that to test Web service methods you should analyze the methods independently of the Web application that uses the methods. In this section I'll show you a basic automated test framework that has proven highly effective on several large projects.

First I created a new VB.NET console application project. Test automation in .NET can be implemented with Windows application and ASP.NET Web application projects, but my colleagues and I generally prefer console applications. I used VB.NET here, but I often use C#, as well. Our first problem is adding plumbing to reference the Web service being tested. To do this, we use the wsdl.exe tool in the .NET Framework SDK. It's a command-line tool, and is invoked as:

>wsdl.exe
http://localhost/WebService1/Service1.asmx?WSDL /l:VB

This command generates all the code needed to connect to the Service1 Web service when using VB.NET. Now we open the resulting Service1.vb file, copy the Imports statements to the top of our console application, and copy the single class named Service1 into the VB Module, just below the Main subroutine. Figure 7 shows the structure of the test automation module at this point.

Imports System

Imports System.ComponentModel

' Other Imports.

 

Module Module1

   Sub Main()

    ' etc.

   End Sub

 

   _

   Public Class Service1

     Inherits _

      System.Web.Services.Protocols.SoapHttpClientProtocol

     Public Sub New()

       MyBase.New()

       Me.Url = "http://localhost/WebService1/Service1.asmx"

     End Sub

    ' Other Subs and Functions generated by wsdl.exe.

     Public Function EndGetTitles( _

       ByVal asyncResult As System.IAsyncResult) _

       As System.Data.DataSet

 

       Dim results()As Object = Me.EndInvoke(asyncResult)

       Return CType(results(0), System.Data.DataSet)

     End Function

   End Class

End Module

Figure 7: The structure of the test automation module.

Now we need to create some test cases. In Visual Studio .NET I added a text file named TestCases.txt to the project and manually created seven test cases:

0001#CountTitles#data#2
0002#CountTitles#computer#5
0003#CountTitles#the#18#deliberate fail
0004#CountTitles#with#2
0005#GetTitles#data#The Busy Executive's Database Guide
0006#GetTitles#data#Foo Bar#deliberate fail
0007#GetTitles#data#Prolonged Data Deprivation: Four Case Studies

The structure of this file is very simple. Each line represents a single test case. The first field is an ID, the second field is the name of the method in the Web service to test, the third field is the input parameter value, the fourth field is the expected result, and the optional fifth field is a comment.

The basic algorithm for our test automation is tied to the structure of our test case file (see Figure 8 for the pseudo-code).

loop

  read a test case line

  parse the test case data

 

  if method-to-test is CountTitles()

    invoke CountTitles() using input

    if actual result = expected result

      result = pass

    else

      result = fail

    log result

 

  if method-to-test is GetTitles()

    invoke GetTitles() using input

    if DataSet contains expected result

      result = pass

    else

       result = fail

    log result

end loop

Figure 8: Pseudo-code for our test automation.

We start by declaring our key variables and objects in the Main subroutine:

Dim FStream As FileStream = _

   New FileStream("..\TestCases.txt", FileMode.Open)

Dim SReader As StreamReader = New StreamReader(FStream)

Dim line As String

Dim tokens()As String

Dim caseid, webMethod, input, expected, actual As String

Dim WebSvc As Service1 = New Service1

Dim pass As Boolean

The purpose of each of these objects should be clear. The tokens variable is an array of type String and will be used to hold each field from a test case line. Notice that we can declare the Service1 Web service as though it were a local object, because we added the wsdl.exe-generated code.

Next we print minimal header information to the screen:

Console.WriteLine( _

  "Starting test run" & Environment.NewLine)

Console.WriteLine("Test ID  WebMethod     Result")

Console.WriteLine( _

  "======================================================")

Now the bulk of the test automation work is performed inside the test case controlled loop (see Figure 9).

line = SReader.ReadLine()

While (line <> Nothing)

  tokens = line.Split("#")

  caseid = tokens(0)

  webMethod = tokens(1)

  input = tokens(2)

  expected = tokens(3)

  pass = False

   Select Case webMethod

     Case "CountTitles"

      ' Test CountTitles()

     Case "GetTitles"

      ' Test GetTitles()

   End Select

  line = SReader.ReadLine()

End While

Figure 9: The bulk of the test automation work is performed inside the test case controlled loop.

After a priming read, we use the String.Split method to break up the test case line and store each string between "#" characters into String array tokens. We could work directly with the tokens array, but it makes the code more readable to transfer values into variables with descriptive names such as caseid and expected. We branch on the value of the webMethod variable with a Select statement, because each Web service method is tested differently.

Let's look at the code that verifies each Web service method under test. The code to test CountTitles is straightforward (see Figure 10).

actual = WebSvc.CountTitles(input).ToString()

If actual = expected Then

  pass = True

End If

Console.Write(caseid & "      " & webMethod.PadRight(14))

If pass = True Then

  Console.WriteLine("Pass")

Else

  Console.WriteLine("FAIL -- Actual = " & actual & _

                    " Expected = " & expected)

End If

Figure 10: Testing the CountTitles method.

We feed the input value pulled from the test case to CountTitles, and cast the return value from type Integer to String, so it can be compared with the expected value. Alternatively, we could have cast the expected value to an Integer using the Integer.Parse method. This is a general issue in test automation: the expected result is typically type String, and some type conversion often must be performed to compare against the actual result. You can either convert the expected result (type String) to match the actual result, or convert the actual result into a String to match the expected result.

The code to test the GetTitles method is slightly more complex (see Figure 11).

Dim DSet As DataSet = WebSvc.GetTitles(input)

Dim i As Integer = 0

While (i <= DSet.Tables("Table").Rows.Count - 1 And _

        pass = False)

  actual = DSet.Tables("Table").Rows(i)("title")

   If actual.IndexOf(expected) <> -1 Then

    pass = True

   End If

  i = i + 1

End While

Console.Write(caseid & "      " & webMethod.PadRight(14))

If pass = True Then

  Console.WriteLine("Pass")

Else

  Console.WriteLine( _

    "FAIL -- No '" & expected & "' in dataset")

End If

Figure 11: The code to test the GetTitles method.

After invoking GetTitles with its test input value, we obtain a DataSet object. We iterate through each row of the DataSet, checking to see if the title field of the row contains the expected result string. If we find the expected string, we short circuit out of the loop. Notice that I used the String.IndexOf method rather than comparing actual and expected results for equality; you must make decisions like this on a method-by-method basis.

Notice that the CountTitles method clearly returns a single value, but that GetTitles returns a DataSet that can be conceptually considered either a single value (the DataSet object) or multiple values (the rows of the DataSet). This introduces the subtle problem of determining what exactly constitutes a correct actual result. In our example, if the string "computer" is input to the GetTitles method, the correct result is a DataSet containing five rows. We can construct a single test case that passes only if all five rows are detected, or five related test cases, each of which checks for one of the five rows. In this example I decided to structure the test automation to check for each row. Ultimately, when you must make test automation design decisions like this, there are few theoretical guiding principles; choose the most practical answer.

Discussion

You can modify and extend the basic framework that tests Web services in many ways. For clarity I used a simple text file to store test cases. Good alternatives are XML storage and SQL storage. Using XML to hold your test cases is particularly appropriate when the test cases are shared across groups. Using SQL to hold test cases is useful when you have many test cases.

The technique in this article displays its output to a command shell. In a production environment, you will probably want to write test results to a text file or a SQL database. Writing to a text file is most appropriate when you're on a relatively short production cycle. Writing results to a SQL database is useful when you're in a long production cycle, because you'll be generating lots of data.

I always add data to the results log in a production environment. At a minimum, you'll want to add counters for the number of cases that pass and that fail. I also like to add timing information for each test case and the overall test run. Timing information can uncover problems in the Web service code that basic pass-fail data misses.

The test automation system presented in this article uses the wsdl.exe tool to create the plumbing code you need for the test automation to connect to the Web service under test. It's possible to add a Web reference to the test automation instead. However, having the plumbing code explicit and visible makes it much easier to track down and fix the inevitable problems with the test automation.

In principle, testing Web service methods is very similar to traditional API (Application Programming Interface) testing. Because Web service methods are remote, however, there are additional connectivity issues. This means you'll want to liberally use .NET try-catch blocks to intercept exceptions. As usual for instructional articles, I removed all error checking in the code presented here. Based on my experience, adding exception handling will double the size of your source code, but is well worth the effort.

One valuable use of the technique presented in this article is to construct Developer Regression Tests (DRTs) for your Web services. DRTs are a sequence of automated tests that are run after you make changes to your application. They're designed to determine if your new code has broken existing functionality, before you accept the code. Other related ways to employ the technique in this article are to construct a sequence of Build Verification Tests (BVTs) that can be used by your Build Team to determine if a new build of your service is minimally functional so that it can be released, and to construct a complete sequence of tests (Daily or Weekly Test Pass) that thoroughly test your service.

The value of automated tests increases as Web applications become more complex. Automated testing has five significant advantages over manual testing:

1) Automated testing is much faster. With automated testing it's feasible to run hundreds of thousands of test cases daily.

2) Automated testing is more accurate. Manual tests are subject to all kinds of human errors, ranging from bad keystrokes to incorrect interpretation of actual vs. expected results.

3) Automated testing is more precise. With manual testing there will be variations in exactly how tests are run between different testers and even for a single tester on different days. Automated tests run the same way every time.

4) Automated tests are more efficient. While automated tests are running, software engineers can be doing tasks that can only be done by humans.

5) As a side effect, the development of automated tests improves an engineer's skill set while manual testing can be mind-numbingly boring.

Conclusion

The ability to create and use .NET Web services provides ASP.NET Web developers with dramatic new design possibilities. But the techniques required to programmatically test Web services are not well known. This article has shown you a simple and effective way to create automated tests that verify the functionality of the methods in Web services. In the days before .NET it was possible to "push the product out the door" with incomplete testing. But with the increased emphasis in quality and security, this is no longer a viable plan. The ability to programmatically test ASP.NET Web services is a valuable addition to your .NET skill set.

Comments

Plain text